CN109284168B - Method and system for separating and managing environment configuration and service data of big data platform - Google Patents
Method and system for separating and managing environment configuration and service data of big data platform Download PDFInfo
- Publication number
- CN109284168B CN109284168B CN201811047918.6A CN201811047918A CN109284168B CN 109284168 B CN109284168 B CN 109284168B CN 201811047918 A CN201811047918 A CN 201811047918A CN 109284168 B CN109284168 B CN 109284168B
- Authority
- CN
- China
- Prior art keywords
- virtual machine
- server
- big data
- image
- container
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45533—Hypervisors; Virtual machine monitors
- G06F9/45558—Hypervisor-specific management and integration aspects
- G06F2009/4557—Distribution of virtual machine instances; Migration and load balancing
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Stored Programmes (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a method for separately managing environment configuration and service data of a big data platform, which comprises the steps of generating a virtual machine mirror image, and sending the virtual machine mirror image to each server through a mirror image version management server; the virtual machine mirror image generates a container through instantiation, the container is operated in a server, and the file of the server is directly mapped to the virtual machine through a file mapping mechanism of the virtual machine, so that the virtual machine directly accesses the file on the server; the invention also provides a system for separating and managing the environment configuration and the service data of the big data platform, so that the light weight of the big data based on the virtualization computing platform is realized.
Description
Technical Field
The invention relates to a method and a system for separating and managing environment configuration and service data of a big data platform.
Background
In a traditional virtual machine, an operating system environment and a service data storage space are both in the virtual machine, that is, data in a hard disk of the whole service is stored in the virtual machine. In an application environment of a server, data is often as high as hundreds of TBs, so a traditional virtual machine image file storage mode is not suitable for large data occupation space and large requirements, and once a server has a failure problem, a file with hundreds of TBs needs to be restored by one image file, which is also a complex and risky operation.
Disclosure of Invention
The technical problem to be solved by the invention is to provide a method and a system for environment configuration and service data separation management of a big data platform, so that light weight and quick iterative switching of a big data configuration environment are realized under the condition of not performing any service data migration, and light weight of big data based on a virtualization computing platform is realized.
One of the present invention is realized by: a method for separating and managing environment configuration and service data of a big data platform comprises the following steps:
step 1, generating a virtual machine image, and sending the virtual machine image to each server through an image version management server;
and 2, generating a container by the virtual machine mirror image through instantiation, operating the container in the server, and directly mapping the file of the server to the virtual machine through a file mapping mechanism of the virtual machine to realize that the virtual machine directly accesses the file on the server.
Further, the generating the virtual machine image in the step 1 is further specifically: instantiating the virtual machine image into a container in the virtual machine of the server, then running the container on the virtual machine of the server, and finally storing the virtual machine as the virtual machine image.
Further, the generating of a container by the virtual machine image through instantiation further specifically includes: the virtual machine image instantiates a container in the virtual machine.
The second invention is realized by the following steps: a big data platform environment configuration and business data separation management system comprises:
the system comprises a generation issuing module, a storage module and a management module, wherein the generation issuing module is used for generating a virtual machine image, the virtual machine image is used for storing system environment configuration data and configuration of a system release version and sending the virtual machine image to each server through an image version management server;
and the operation module generates a container by instantiating the virtual machine mirror image, operates the container in the server, and directly maps the file of the server to the virtual machine through a file mapping mechanism of the virtual machine so as to realize that the virtual machine directly accesses the file on the server.
Further, the generating the virtual machine image in the generating and issuing module is further specifically: instantiating the virtual machine image into a container in the virtual machine of the server, then running the container on the virtual machine of the server, and finally storing the virtual machine as the virtual machine image.
Further, the generating of a container by the virtual machine image through instantiation further specifically includes: the virtual machine image instantiates a container in the virtual machine.
The invention has the following advantages:
1) Lightweight large data cluster mirroring: big data cluster configuration and big data service configuration are separated, and only configuration files of big data are reserved in cluster mirror images, so that light management of mirror image files is achieved.
2) Mirror image management big data configuration version: the big data management mode of the original script plus configuration file is changed, the mirror image management big data configuration is realized, and the iterative update of the big data configuration file is realized by issuing a mirror image version through the mirror image server.
3) Configuration and environmental consistency of big data platforms: the system of the whole big data cluster runs in a container generated by the mirror image of the virtual machine of the same version, the environmental consistency of the whole cluster is realized, and the problem of the difference of cluster servers in the operation and maintenance of the big data in the past is avoided.
Drawings
The invention will be further described with reference to the following examples with reference to the accompanying drawings.
FIG. 1 is a flow chart of the method of the present invention.
Detailed Description
As shown in fig. 1, the method for separating and managing environment configuration and service data of a big data platform of the present invention includes:
step 1, instantiating a virtual machine image into a container in a virtual machine of a server, then operating the container on the virtual machine of the server, finally storing the virtual machine image as the virtual machine image, and sending the virtual machine image to each server through an image version management server;
step 2, instantiating the virtual machine mirror image in the virtual machine to generate a container, operating the container in the server, and directly mapping the file of the server to the virtual machine through a file mapping mechanism of the virtual machine to realize that the virtual machine directly accesses the file on the server.
The invention relates to a big data platform environment configuration and service data separation management system, which comprises:
the method comprises the steps that an issuing module is generated, a virtual machine image is instantiated into a container in a virtual machine of a server, then the container is operated on the virtual machine of the server, finally the virtual machine is stored as the virtual machine image, and the virtual machine image is sent to each server through an image version management server;
the virtual machine mirror image is instantiated in the virtual machine to generate a container, the container is operated in the server, and the file of the server is directly mapped to the virtual machine through a file mapping mechanism of the virtual machine, so that the virtual machine directly accesses the file on the server.
One embodiment of the present invention:
the invention realizes that the virtual machine ISO can directly access the host machine hard disk directory space, the data of the big data platform is left in the host machine, the system environment of big data operation, the big data configuration file and the like are put on the virtual machine platform, the configuration version of the big data platform can be managed as long as the virtual machine image file is managed, the configuration version of the big data platform is separated from the big data service data, and the hierarchical management is realized. The method realizes light-weight rapid iterative switching of a big data configuration environment under the condition of not carrying out any business data migration, realizes light weight of big data based on a virtualization computing platform, and can rapidly carry out switching iterative upgrading of a virtualization container.
The scheme structure is as follows: mirror image layering realization, mirror image and container conversion and mirror image management big data configuration.
The method mainly comprises the following steps:
a, mirror layering implementation
Layering a Linux kernel and a system release version: in a pure Linux system environment, the kernel-related bottom-layer driver configuration and the dependent environment package occupy most of the system files, the system issues version configuration files, and the system kernel occupies hardware resources of a host machine after the system is started.
Layering of service data and big data configuration: the file of the host machine is directly mapped to the virtual machine by realizing a file mapping mechanism of the virtual machine, so that the virtual machine can directly access the file of the host machine, the virtual machine mirror image does not need to store user service data of a big data platform, and the virtual machine mirror image only needs to store system environment configuration data of the big data, so that the layering of the service data of the user big data and the system environment configuration is realized.
Mirror image and container conversion
The mirror image can generate a container to be operated in the host machine through instantiation, the file with the static mirror image is issued by the mirror image management server, the mirror image can be operated on the host machine after the virtual machine of the host machine instantiates the mirror image into the container, and the container operated in the virtual machine can be stored as the mirror image and stored in the mirror image server.
Mirror image management big data configuration
Saving the configuration file as a mirror image: configuration parameters and the like of big data are directly configured in an operating container, and then the container is saved as a mirror image, so that the configuration file can be saved in the mirror image.
Iterative management of big data configuration: the distribution of the configuration files of the large-scale big data clusters can be realized by issuing the updated mirror image through the mirror image version management server, so that the configuration files take effect in each cluster of the big data, the consistency of the environment and the configuration files of the whole big data cluster is ensured, and the problem of system difference is avoided.
Although specific embodiments of the invention have been described above, it will be understood by those skilled in the art that the specific embodiments described are illustrative only and are not limiting upon the scope of the invention, and that equivalent modifications and variations can be made by those skilled in the art without departing from the spirit of the invention, which is to be limited only by the appended claims.
Claims (6)
1. A big data platform environment configuration and service data separation management method is characterized in that: the method comprises the following steps:
step 1, generating a virtual machine image, and sending the virtual machine image to each server through an image version management server; the virtual machine mirror image only stores system environment configuration data of big data;
step 2, generating a container by the virtual machine mirror image through instantiation, operating the container in a server, and directly mapping the file of the server to the virtual machine through a file mapping mechanism of the virtual machine to realize that the virtual machine directly accesses the file on the server;
step 3, iterative management of big data configuration: the virtual machine mirror image which is subjected to the updating configuration is issued through the mirror image version management server, the distribution of the configuration file of the big data cluster is realized, the configuration file is enabled to take effect in each cluster of the big data, and the consistency of the environment and the configuration file of the whole big data cluster is ensured.
2. The big data platform environment configuration and service data separation management method according to claim 1, wherein: the generating the virtual machine image in the step 1 is further specifically: instantiating the virtual machine image into a container in the virtual machine of the server, then running the container on the virtual machine of the server, and finally storing the virtual machine as the virtual machine image.
3. The big data platform environment configuration and service data separation management method according to claim 1, wherein: the generating of a container by the virtual machine image through instantiation further specifically comprises: the virtual machine image instantiates a container in the virtual machine.
4. A big data platform environment configuration and service data separation management system is characterized in that: the method comprises the following steps:
the system comprises a generation issuing module, a storage module and a management module, wherein the generation issuing module is used for generating a virtual machine image, the virtual machine image is used for storing system environment configuration data and configuration of a system release version and sending the virtual machine image to each server through an image version management server; the virtual machine mirror image only stores system environment configuration data of big data;
the operation module generates a container by instantiating the virtual machine mirror image, operates the container in the server, and directly maps the file of the server to the virtual machine through a file mapping mechanism of the virtual machine so as to realize that the virtual machine directly accesses the file on the server;
an iteration management module: the virtual machine mirror image which is subjected to the updating configuration is issued through the mirror image version management server, the distribution of the configuration file of the big data cluster is realized, the configuration file is enabled to take effect in each cluster of the big data, and the consistency of the environment and the configuration file of the whole big data cluster is ensured.
5. The big data platform environment configuration and service data separation management system according to claim 4, wherein: the generating of the virtual machine image in the generating and issuing module further includes: instantiating the virtual machine image into a container in the virtual machine of the server, then running the container on the virtual machine of the server, and finally storing the virtual machine as the virtual machine image.
6. The big data platform environment configuration and service data separation management system according to claim 4, wherein: the generating of a container by the virtual machine image through instantiation further specifically comprises: the virtual machine image is instantiated in a virtual machine to generate a container.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811047918.6A CN109284168B (en) | 2018-09-10 | 2018-09-10 | Method and system for separating and managing environment configuration and service data of big data platform |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811047918.6A CN109284168B (en) | 2018-09-10 | 2018-09-10 | Method and system for separating and managing environment configuration and service data of big data platform |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109284168A CN109284168A (en) | 2019-01-29 |
| CN109284168B true CN109284168B (en) | 2022-12-06 |
Family
ID=65184319
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811047918.6A Active CN109284168B (en) | 2018-09-10 | 2018-09-10 | Method and system for separating and managing environment configuration and service data of big data platform |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109284168B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110928639B (en) * | 2019-10-22 | 2022-12-02 | 西安雷风电子科技有限公司 | Windows virtualization mirror image hierarchical management method and device |
| CN110830571A (en) * | 2019-11-05 | 2020-02-21 | 许继集团有限公司 | Business data backup and extraction method and computer readable medium |
| CN112948062B (en) * | 2021-05-13 | 2021-07-13 | 鹏城实验室 | Transparent transmission method and device for device file and computer storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102185928A (en) * | 2011-06-01 | 2011-09-14 | 广州杰赛科技股份有限公司 | Method for creating virtual machine in cloud computing system and cloud computing system |
| CN107547250A (en) * | 2017-06-26 | 2018-01-05 | 新华三云计算技术有限公司 | The method and apparatus that database is disposed in cloud computing management platform |
Family Cites Families (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8181174B2 (en) * | 2007-12-28 | 2012-05-15 | Accenture Global Services Limited | Virtual machine configuration system |
| US8438349B2 (en) * | 2009-08-21 | 2013-05-07 | Symantec Corporation | Proxy backup of virtual disk image files on NAS devices |
| US20170123996A1 (en) * | 2015-11-02 | 2017-05-04 | Microsoft Technology Licensing, Llc | Direct Mapped Files in Virtual Address-Backed Virtual Machines |
| US9766915B1 (en) * | 2016-03-23 | 2017-09-19 | Parallels IP Holdings GmbH | Method for creation of application containers inside OS containers |
| CN106199696B (en) * | 2016-06-29 | 2019-01-18 | 中国石油天然气股份有限公司 | seismic data processing system and method |
| US10901966B2 (en) * | 2016-08-29 | 2021-01-26 | Vmware, Inc. | Synchronizing configurations for container hosted applications |
| CN106385449B (en) * | 2016-09-13 | 2019-01-29 | 哈尔滨工程大学 | A kind of isomeric group services migrating transferring system and middle shifting method based on container |
| CN108111575B (en) * | 2017-11-29 | 2020-12-01 | 北京京航计算通讯研究所 | Extensible client and virtual machine data interaction system |
| CN107872467A (en) * | 2017-12-26 | 2018-04-03 | 中国联合网络通信集团有限公司 | Honey jar active defense method and honey jar Active Defending System Against based on Serverless frameworks |
-
2018
- 2018-09-10 CN CN201811047918.6A patent/CN109284168B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102185928A (en) * | 2011-06-01 | 2011-09-14 | 广州杰赛科技股份有限公司 | Method for creating virtual machine in cloud computing system and cloud computing system |
| CN107547250A (en) * | 2017-06-26 | 2018-01-05 | 新华三云计算技术有限公司 | The method and apparatus that database is disposed in cloud computing management platform |
Non-Patent Citations (1)
| Title |
|---|
| 快速部署虚拟机的新方法;万林;《中国教育网络》;20110405(第04期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109284168A (en) | 2019-01-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12118341B2 (en) | Conversion and restoration of computer environments to container-based implementations | |
| US11630689B2 (en) | Image subunit based guest scheduling | |
| US20210048998A1 (en) | Container-based server environments | |
| US8434081B2 (en) | Storage manager for virtual machines with virtual storage | |
| US11093148B1 (en) | Accelerated volumes | |
| EP3039575B1 (en) | Scalable distributed storage architecture | |
| US9946578B2 (en) | Managing the persistent data of a pre-installed application in an elastic virtual machine instance | |
| CN109284168B (en) | Method and system for separating and managing environment configuration and service data of big data platform | |
| US20220129355A1 (en) | Creation of virtual machine packages using incremental state updates | |
| US20120005672A1 (en) | Image management for virtual machine instances and associated virtual storage | |
| US20120047107A1 (en) | System and method for implementing on demand cloud database | |
| US20130262923A1 (en) | Efficient application management in a cloud with failures | |
| CN102693230B (en) | For the file system of storage area network | |
| US8813076B2 (en) | Virtual machine updates | |
| CN114461333A (en) | Container mirror creation and deployment | |
| CN116508001A (en) | Unloading a container execution environment | |
| US20230342134A1 (en) | Hybrid approach to performing a lazy pull of container images | |
| US20220318042A1 (en) | Distributed memory block device storage | |
| KR102231358B1 (en) | Single virtualization method and system for HPC cloud service | |
| US20230066840A1 (en) | Efficiently providing a guest context access to file content at a host context | |
| US11797358B2 (en) | Method and system for performing application programming interface calls between heterogeneous applications and cloud service providers | |
| US20220236893A1 (en) | System and method for distributed deduplication in a composed system | |
| WO2023185570A1 (en) | Storage driver for managing multiple layer file system on cloud | |
| CN117978802A (en) | Cloud resource management system and deployment method thereof | |
| CN118451400A (en) | Method and system for performing predictive compounding on a compounding information handling system using telemetry data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 350000 21 / F, building 5, f District, Fuzhou Software Park, 89 software Avenue, Gulou District, Fuzhou City, Fujian Province Applicant after: FUJIAN SINOREGAL SOFTWARE CO.,LTD. Address before: Floor 20-21, building 5, area F, Fuzhou Software Park, 89 software Avenue, Gulou District, Fuzhou City, Fujian Province 350000 Applicant before: FUJIAN SINOREGAL SOFTWARE CO.,LTD. |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |