CN109284204A - A kind of big data platform O&M method and system calculated based on virtualization - Google Patents
A kind of big data platform O&M method and system calculated based on virtualization Download PDFInfo
- Publication number
- CN109284204A CN109284204A CN201811047942.XA CN201811047942A CN109284204A CN 109284204 A CN109284204 A CN 109284204A CN 201811047942 A CN201811047942 A CN 201811047942A CN 109284204 A CN109284204 A CN 109284204A
- Authority
- CN
- China
- Prior art keywords
- virtual machine
- server
- image
- big data
- machine image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/14—Error detection or correction of the data by redundancy in operation
- G06F11/1402—Saving, restoring, recovering or retrying
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1095—Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
Abstract
The present invention provides a kind of big data platform O&M method calculated based on virtualization, establishes virtual machine image, the virtual machine image includes system configuration file and user environment variable；Virtual machine image is saved in mirror image warehouse server after generating, and manages image release according to the time sequencing that virtual machine image generates；According to demand, the virtual machine image of corresponding version is sent on server, server runs the virtual machine image；The present invention also provides a kind of big data platform operational systems calculated based on virtualization, are convenient for user maintenance.
Description
Technical field
The present invention relates to a kind of big data platform O&M methods and system calculated based on virtualization.
Background technique
Existing big data O&M is that the SSH based on server ip address remotely connects server, passes through automatized script
Order management server is executed, but often arranges hundreds and thousands of servers in big data cluster, if using
Manual operations is safeguarded, then under its efficiency is very low, difficulty is also very big, wherein a large amount of O&M script, disorderly and unsystematic, it is interior
Bulk density is multiple, and quality is difficult to ensure, finally leaves hidden danger to failure, all not to the definition and management of existing net expecting state
Existing net state is all the product that script accumulates over a long period, and causes server state to drift about, and then leave hidden danger to service stability.
Summary of the invention
The technical problem to be solved in the present invention is to provide a kind of big data platform O&M method calculated based on virtualization
And system, it realizes from traditional distributed script execution, becomes distributed container replacement, safeguarded convenient for user.
One of present invention is achieved in that a kind of big data platform O&M method calculated based on virtualization, comprising:
Step 1 establishes virtual machine image, and the virtual machine image includes system configuration file and user environment variable；
Step 2, virtual machine image are saved in mirror image warehouse server after generating, and the time generated according to virtual machine image is suitable
Sequence manages image release；
The virtual machine image of corresponding version is sent on server by step 3, according to demand, and server runs the void
Quasi- machine mirror image.
Further, further include the journal file of step 4, the entire big data server cluster of acquisition, pass through log analysis
There is abnormal server, the server info for occurring abnormal in log analysis is fed back.
Further, further include step 5, the abnormal server progress algorithm analysis for feedback, judge whether to need to do
The operation of image recovery then notifies mirror image warehouse server to update normal mirror to abnormal if needing to do image recovery operation
Server, which is done, to be updated.
The two of the present invention are achieved in that a kind of big data platform operational system calculated based on virtualization, comprising:
Module is established, virtual machine image is established, the virtual machine image includes that system configuration file and user environment become
Amount；
Management module, virtual machine image generate after be saved in mirror image warehouse server, according to virtual machine image generate when
Between sequence management image release；
The virtual machine image of corresponding version is sent on server by update module according to demand, described in server operation
Virtual machine image.
Further, further include log module, acquire the journal file of entire big data server cluster, pass through log point
Existing abnormal server is precipitated, the server info for occurring abnormal in log analysis is fed back.
Further, further include recovery module, carry out algorithm analysis for the abnormal server of feedback, judge whether to need
The operation of image recovery is done, if needing to do image recovery operation, then mirror image warehouse server is notified to update normal mirror to different
Normal server, which is done, to be updated.
Further, described to establish virtual machine image further specifically: by virtual machine mirror in the virtual machine of server
As being instantiated as container, is operated on the virtual machine of server later, the virtual machine is finally saved as into virtual machine image.
The present invention has the advantage that
1) big data platform version management is realized by virtualization: system environments, configuration, data point is realized by virtualization
From the fast quick-recovery and upgrading iteration for realizing big data cluster.
2) fault recovery automated: by the automatic operational system of big data platform, service of breaking down in cluster is realized
It can be automatically performed the image recovery of malfunctioning node when device node, realize the automatic recovery of the malfunctioning node of big data platform.
3) it saves O&M cost of labor: realizing and script management configuration is converted into image release management configuration, and pass through
Automation operational system of the invention can greatly reduce the cost of manpower intervention, and automation and intelligence largely may be implemented
The O&M target of change.
Detailed description of the invention
The present invention is further illustrated in conjunction with the embodiments with reference to the accompanying drawings.
Fig. 1 is the method for the present invention execution flow chart.
Specific embodiment
As shown in Figure 1, the present invention is based on the big data platform O&M methods that virtualization calculates, comprising:
Step 1 establishes virtual machine image, and the virtual machine image includes system configuration file and user environment variable；
Step 2, virtual machine image are saved in mirror image warehouse server after generating, and the time generated according to virtual machine image is suitable
Sequence manages image release；
The virtual machine image of corresponding version is sent on server by step 3, according to demand, and server runs the void
Quasi- machine mirror image.
The journal file of step 4, the entire big data server cluster of acquisition, occurs abnormal service by log analysis
Device feeds back the server info for occurring abnormal in log analysis.
Step 5, the abnormal server progress algorithm analysis for feedback, judge whether the operation for needing to do image recovery,
If needing to do image recovery operation, then notify mirror image warehouse server updates normal mirror and does to abnormal server to update.
The present invention is based on the big data platform operational systems that virtualization calculates, comprising:
Module is established, virtual machine image is established, the virtual machine image includes that system configuration file and user environment become
Amount；
Management module, virtual machine image generate after be saved in mirror image warehouse server, according to virtual machine image generate when
Between sequence management image release；
The virtual machine image of corresponding version is sent on server by update module according to demand, described in server operation
Virtual machine image.
Log module acquires the journal file of entire big data server cluster, occurs abnormal clothes by log analysis
Business device, the server info for occurring abnormal in log analysis is fed back.
Recovery module carries out algorithm analysis for the abnormal server of feedback, judges whether the behaviour for needing to do image recovery
Make, if needing to do image recovery operation, then notifies mirror image warehouse server updates normal mirror and does to abnormal server to update.
Virtual machine image generates a container by instantiation, in the server by container operation, passes through virtual machine
File Mapping mechanism, the file of server is mapped directly into virtual machine, realizes that virtual machine directly accesses the text on server
Part.
A kind of specific embodiment of the present invention:
Scenario-frame: mirror image generation, the version management of mirror image warehouse, log monitoring system, virtualization based on container technique
The automatic operational system of big data platform.
Key step:
A: mirror image generates server
1) based on the virtualization of container technique: by by linux system kernel bottom layer operating system and system configuration file
Separation, the virtual machine image that the present invention realizes is mainly system configuration file and user environment variable etc., belongs to lightweight
Virtualization.I.e. bottom linux kernel of host etc. is used as container, and the virtual machine of each version configuration operates in this bottom appearance
Among device, there was only the configurations and system environment variable of user in virtual machine.
2) lightweight mirror is as file generated: the data volume of big data platform is very huge, and image file of the present invention, which is realized, to be used
The separation of family business datum and system configuration environmental variance, mirror image of the invention does not include the business datum of big data platform, real
Existing light-weighted image file generates, and the big data platform system image for the second grade that lightweight mirror picture can be done updates, and realizes big
Data platform configuration version is switched fast, and traffic affecting does not operate normally.
B: mirror image warehouse server
1) image release management: mirror image is saved in mirror image warehouse server after generating, this server carries out image release
Management manages image release according to the time sequencing of generation；
2) image release issues: big data platform system version may be implemented by the rollback of mirror image manager version to restore
System, or realize by version management the system and configuration of the upgrading big data platform of pre-configured system image.
C: log monitoring system
1) log collection is analyzed: acquiring the journal file of entire big data server cluster, occurs by log analysis different
Normal server.
2) abnormal feedback: will occur the information such as abnormal server ID in log analysis and feed back to big data platform transporting automatically
Maintain system.
D: the automatic operational system of big data platform
Abnormal server handles decision: carrying out algorithm analysis for the abnormal server of log server feedback, judgement is
The no operation for needing to do image recovery.Next notice mirror image warehouse server is updated into normal mirror version to abnormal server
Version updating is done, because big data platform sheet, as HA mode, wherein a server does shutdown switching not within the same time
Influence service operation；If not needing image recovery operation, then carry out task processing is not needed.
The configuration management of normal mirror file: it the image file of normal operation and can be done relevant big on this server
The verifying (verifying configuration whether as actual demand for operator) of data configuration file, may be implemented subsequent configuration file
Centralized management, server editor configuration, can entire cluster realize the upgrading of big data configuration file.
Although specific embodiments of the present invention have been described above, those familiar with the art should be managed
Solution, we are merely exemplary described specific embodiment, rather than for the restriction to the scope of the present invention, it is familiar with this
The technical staff in field should be covered of the invention according to modification and variation equivalent made by spirit of the invention
In scope of the claimed protection.
Claims (8)
1. a kind of big data platform O&M method calculated based on virtualization, it is characterised in that: include:
Step 1 establishes virtual machine image, and the virtual machine image includes system configuration file and user environment variable；
Step 2, virtual machine image are saved in mirror image warehouse server after generating, the time sequencing pipe generated according to virtual machine image
Manage image release；
The virtual machine image of corresponding version is sent on server by step 3, according to demand, and server runs the virtual machine
Mirror image.
2. a kind of big data platform O&M method calculated based on virtualization according to claim 1, it is characterised in that: also
Journal file including step 4, the entire big data server cluster of acquisition, occurs abnormal server by log analysis, will
Occur abnormal server info in log analysis to be fed back.
3. a kind of big data platform O&M method calculated based on virtualization according to claim 2, it is characterised in that: also
Algorithm analysis is carried out including step 5, for the abnormal server of feedback, judges whether the operation for needing to do image recovery, if
It needs to do image recovery operation, then notifies mirror image warehouse server updates normal mirror and does to abnormal server to update.
4. a kind of big data platform O&M method calculated based on virtualization according to claim 1, it is characterised in that: institute
State that establish virtual machine image further specifically: virtual machine image is instantiated as container in the virtual machine of server, later
It operates on the virtual machine of server, the virtual machine is finally saved as into virtual machine image.
5. a kind of big data platform operational system calculated based on virtualization, it is characterised in that: include:
Module is established, virtual machine image is established, the virtual machine image includes system configuration file and user environment variable；
Management module, virtual machine image are saved in mirror image warehouse server after generating, and the time generated according to virtual machine image is suitable
Sequence manages image release；
The virtual machine image of corresponding version is sent on server by update module according to demand, and server operation is described virtual
Machine mirror image.
6. a kind of big data platform operational system calculated based on virtualization according to claim 5, it is characterised in that: also
Including log module, the journal file of entire big data server cluster is acquired, occurs abnormal server by log analysis,
The server info for occurring abnormal in log analysis is fed back.
7. a kind of big data platform operational system calculated based on virtualization according to claim 6, it is characterised in that: also
Including recovery module, algorithm analysis is carried out for the abnormal server of feedback, judges whether the operation for needing to do image recovery, if
It is to need to do image recovery operation, then notifies mirror image warehouse server updates normal mirror and does to abnormal server to update.
8. a kind of big data platform operational system calculated based on virtualization according to claim 5, it is characterised in that: institute
State that establish virtual machine image further specifically: virtual machine image is instantiated as container in the virtual machine of server, later
It operates on the virtual machine of server, the virtual machine is finally saved as into virtual machine image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811047942.XA CN109284204B (en) | 2018-09-10 | 2018-09-10 | Big data platform operation and maintenance method and system based on virtualization computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811047942.XA CN109284204B (en) | 2018-09-10 | 2018-09-10 | Big data platform operation and maintenance method and system based on virtualization computing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109284204A true CN109284204A (en) | 2019-01-29 |
| CN109284204B CN109284204B (en) | 2022-10-25 |
Family
ID=65183901
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811047942.XA Active CN109284204B (en) | 2018-09-10 | 2018-09-10 | Big data platform operation and maintenance method and system based on virtualization computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109284204B (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111061540A (en) * | 2019-11-27 | 2020-04-24 | 北京计算机技术及应用研究所 | Application virtualization method and system based on container technology |
| CN111953788A (en) * | 2020-08-17 | 2020-11-17 | 浪潮云信息技术股份公司 | Large-scale cloud platform |
Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101009589A (en) * | 2006-01-26 | 2007-08-01 | 腾讯科技（深圳）有限公司 | A method and system for software upgrade |
| CN101540799A (en) * | 2009-04-23 | 2009-09-23 | 深圳市融创天下科技发展有限公司 | Mobile terminal software upgrading method |
| CN101594387A (en) * | 2009-06-29 | 2009-12-02 | 北京航空航天大学 | The virtual cluster deployment method and system |
| US20090319580A1 (en) * | 2008-06-23 | 2009-12-24 | Dean Lorenz | Hypervisor Service to Provide Image Version Control Support |
| US20130139183A1 (en) * | 2011-11-28 | 2013-05-30 | Wyse Technology Inc. | Creation or installation of a disk image for a target device having one of a plurality of hardware platforms |
| US20140201572A1 (en) * | 2013-01-13 | 2014-07-17 | International Business Machines Corporation | Provisioning virtual environments based on policies for troubleshooting purposes |
| CN105550012A (en) * | 2015-12-07 | 2016-05-04 | 国云科技股份有限公司 | Method for custom recovery of malfunctioning virtual machine |
| CN105912382A (en) * | 2016-04-07 | 2016-08-31 | 浪潮电子信息产业股份有限公司 | Mirror image management device, system and method |
| CN106199696A (en) * | 2016-06-29 | 2016-12-07 | 中国石油天然气股份有限公司 | Earthquake data processing system and method |
| CN106528224A (en) * | 2016-11-03 | 2017-03-22 | 腾讯科技（深圳）有限公司 | Content updating method and system for Docker container, and server |
| CN107066296A (en) * | 2017-03-31 | 2017-08-18 | 北京奇艺世纪科技有限公司 | The method for cleaning and device of mirror image in a kind of clustered node |
| CN107196803A (en) * | 2017-05-31 | 2017-09-22 | 中国人民解放军信息工程大学 | The dynamic generation and maintaining method of isomery cloud main frame |
| CN107294772A (en) * | 2017-05-23 | 2017-10-24 | 甘肃万维信息技术有限责任公司 | One kind combines Docker and realizes dynamic management and monitoring service system |
| CN107395762A (en) * | 2017-08-30 | 2017-11-24 | 四川长虹电器股份有限公司 | A kind of application service based on Docker containers accesses system and method |
-
2018
- 2018-09-10 CN CN201811047942.XA patent/CN109284204B/en active Active
Patent Citations (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101009589A (en) * | 2006-01-26 | 2007-08-01 | 腾讯科技（深圳）有限公司 | A method and system for software upgrade |
| US20090319580A1 (en) * | 2008-06-23 | 2009-12-24 | Dean Lorenz | Hypervisor Service to Provide Image Version Control Support |
| CN101540799A (en) * | 2009-04-23 | 2009-09-23 | 深圳市融创天下科技发展有限公司 | Mobile terminal software upgrading method |
| CN101594387A (en) * | 2009-06-29 | 2009-12-02 | 北京航空航天大学 | The virtual cluster deployment method and system |
| US20130139183A1 (en) * | 2011-11-28 | 2013-05-30 | Wyse Technology Inc. | Creation or installation of a disk image for a target device having one of a plurality of hardware platforms |
| US20140201572A1 (en) * | 2013-01-13 | 2014-07-17 | International Business Machines Corporation | Provisioning virtual environments based on policies for troubleshooting purposes |
| CN105550012A (en) * | 2015-12-07 | 2016-05-04 | 国云科技股份有限公司 | Method for custom recovery of malfunctioning virtual machine |
| CN105912382A (en) * | 2016-04-07 | 2016-08-31 | 浪潮电子信息产业股份有限公司 | Mirror image management device, system and method |
| CN106199696A (en) * | 2016-06-29 | 2016-12-07 | 中国石油天然气股份有限公司 | Earthquake data processing system and method |
| CN106528224A (en) * | 2016-11-03 | 2017-03-22 | 腾讯科技（深圳）有限公司 | Content updating method and system for Docker container, and server |
| CN107066296A (en) * | 2017-03-31 | 2017-08-18 | 北京奇艺世纪科技有限公司 | The method for cleaning and device of mirror image in a kind of clustered node |
| CN107294772A (en) * | 2017-05-23 | 2017-10-24 | 甘肃万维信息技术有限责任公司 | One kind combines Docker and realizes dynamic management and monitoring service system |
| CN107196803A (en) * | 2017-05-31 | 2017-09-22 | 中国人民解放军信息工程大学 | The dynamic generation and maintaining method of isomery cloud main frame |
| CN107395762A (en) * | 2017-08-30 | 2017-11-24 | 四川长虹电器股份有限公司 | A kind of application service based on Docker containers accesses system and method |
Non-Patent Citations (4)
| Title |
|---|
| C PRAKASH 等: "Deterministic Container Resource Management in Derivative Clouds", 《2018 IEEE INTERNATIONAL CONFERENCE ON CLOUD ENGINEERING (IC2E)》 * |
| CHUNG PAN TANG等: "CloudVS: Enabling version control for virtual machines in an open-source cloud under commodity settings", 《2012 IEEE NETWORK OPERATIONS AND MANAGEMENT SYMPOSIUM》 * |
| 易升海等: "浅析Docker容器技术的发展前景", 《电信工程技术与标准化》 * |
| 金永霞 等: "云计算实验平台虚拟机镜像的管理与维护", 《实验技术与管理》 * |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111061540A (en) * | 2019-11-27 | 2020-04-24 | 北京计算机技术及应用研究所 | Application virtualization method and system based on container technology |
| CN111061540B (en) * | 2019-11-27 | 2023-05-23 | 北京计算机技术及应用研究所 | Application virtualization method and system based on container technology |
| CN111953788A (en) * | 2020-08-17 | 2020-11-17 | 浪潮云信息技术股份公司 | Large-scale cloud platform |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109284204B (en) | 2022-10-25 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11121921B2 (en) | Dynamic auto-configuration of multi-tenant PaaS components | |
| EP2672668B1 (en) | Facilitating the operation of a virtual network by predicting a failure | |
| US10114665B2 (en) | Communication node upgrade system and method for a communication network | |
| CN112600891B (en) | Information physical fusion-based edge cloud cooperative system and working method | |
| CN110011869B (en) | Controller apparatus, method, and computer-readable storage medium | |
| CN109189680B (en) | A kind of system and method for application publication and configuration | |
| US20180357130A1 (en) | Method and system to discover and manage distributed applications in virtualization environments | |
| US8054763B2 (en) | Migration of switch in a storage area network | |
| CN103490941A (en) | Real-time monitoring on-line configuration method in cloud computing environment | |
| CN102437938A (en) | Large-scale network monitoring oriented virtual deployment system and method | |
| JP2001209622A (en) | System for managing constitution of data processor | |
| CN110134518A (en) | A kind of method and system improving big data cluster multinode high application availability | |
| US10592657B2 (en) | Automated secure disposal of hardware components | |
| CN109284204A (en) | A kind of big data platform O&M method and system calculated based on virtualization | |
| US20200412619A1 (en) | Network System, Topology Management Method, and Program | |
| CN108959009A (en) | A kind of server failure analysis method and its fail analysis device | |
| CN107888438B (en) | Flow table technology-based system for automatically sensing and adapting to cloud environment change | |
| CN109104333B (en) | GIT-based distributed cluster synchronization method and device | |
| US11893411B2 (en) | System and method for resource optimized intelligent product notifications | |
| US20240103824A1 (en) | Server management apparatus and server management method | |
| CN113824801A (en) | Intelligent integration terminal unified access management component system | |
| US10812379B2 (en) | Baggage messaging handling systems and methods | |
| WO2020139072A1 (en) | A method of migrating virtual machines | |
| CN110401559A (en) | A kind of server restorative procedure and device | |
| US20240103947A1 (en) | Server management apparatus and server management method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information | ||
| CB02 | Change of applicant information |
Address after: 350000 21 / F, building 5, f District, Fuzhou Software Park, 89 software Avenue, Gulou District, Fuzhou City, Fujian Province Applicant after: FUJIAN SINOREGAL SOFTWARE CO.,LTD. Address before: Floor 20-21, building 5, area F, Fuzhou Software Park, 89 software Avenue, Gulou District, Fuzhou City, Fujian Province 350000 Applicant before: FUJIAN SINOREGAL SOFTWARE CO.,LTD. |
|
| GR01 | Patent grant | ||
| GR01 | Patent grant |