CN109190381B - Method for detecting hadoop security vulnerability - Google Patents
Method for detecting hadoop security vulnerability Download PDFInfo
- Publication number
- CN109190381B CN109190381B CN201811040881.4A CN201811040881A CN109190381B CN 109190381 B CN109190381 B CN 109190381B CN 201811040881 A CN201811040881 A CN 201811040881A CN 109190381 B CN109190381 B CN 109190381B
- Authority
- CN
- China
- Prior art keywords
- scanning
- hadoop
- detection
- task
- security
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/57—Certifying or maintaining trusted computer platforms, e.g. secure boots or power-downs, version controls, system software checks, secure updates or assessing vulnerabilities
- G06F21/577—Assessing vulnerabilities and evaluating computer system security
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/50—Monitoring users, programs or devices to maintain the integrity of platforms, e.g. of processors, firmware or operating systems
- G06F21/55—Detecting local intrusion or implementing counter-measures
- G06F21/56—Computer malware detection or handling, e.g. anti-virus arrangements
- G06F21/562—Static detection
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Virology (AREA)
- Storage Device Security (AREA)
- Collating Specific Patterns (AREA)
Abstract
The invention relates to a detection method for hadoop security holes. Setting a hadoop security vulnerability database, and adding an object scanned by a hadoop core component; generating a scanning task after adding a scanning strategy and configuration to a scanning object; calling a scanning engine to scan the created scanning task; the scanning engine automatically crawls directories and files of the scanned objects and simultaneously realizes the detection of crawling and crawling of the scanned objects; when the scanning task and the scanning strategy do not need to be modified, scanning is executed until the scanning task is finished; otherwise, the scanning task may be stopped to modify the scanning task and the scanning strategy, and then the scanning task is executed again: and after the scanning task is finished, outputting a scanning result report according to a scanning report template built in the system. The invention can monitor the system in real time, scan the core components of the system and generate the report at the first time, thereby reducing the damage of the vulnerability to the system and ensuring the security of enterprise data.
Description
Technical Field
The invention belongs to the field of big data, and particularly relates to a method for detecting hadoop security vulnerabilities.
Background
Hadoop consists of multiple components, so the secure representation of a Hadoop cluster ensures the security of each component, which makes Hadoop cluster security a complex task.
The enterprise data contains key information related to sales, customer information, human resources and the like, and is safely stored in information management systems such as ERP, CRM and general ledger system. Over the last decade, various data security incidents have resulted in billions of dollars of loss, also driving enterprise data security technologies toward maturity. As the service industry grows and matures, most systems are open to suppliers most of the time to process critical customer information. Thus, a number of relevant standards for security and privacy protection have been developed, including HIPAA, HITECH, PCI, SOX, ISO, and COBIT. Service providers are therefore required to strictly adhere to these standards to fully protect customer data. This also provides powerful data security safeguards for service providers and customers within the enterprise, without tolerating any data security violations. In the development process of the last eight years, Hadoop has become mature day by day, enterprises begin to adopt Hadoop to meet the requirement of big data processing, and any data security accident can cause the enterprises to lose trust of Hadoop ecosystems. Unless business organizations are fully trusted with the Hadoop ecosystem, they never risk adopting the big data technology, so the key to success or failure of big data items is how to ensure the security of data by the data ecosystem, and it is very necessary to check and clean bugs in time.
Disclosure of Invention
The invention aims to provide a method for detecting hadoop security vulnerabilities, which can monitor a system in real time, scan core components of the system and generate reports at the first time, reduce damage of vulnerabilities to the system and guarantee security of enterprise data.
In order to achieve the purpose, the technical scheme of the invention is as follows: a method for detecting hadoop security vulnerabilities comprises the following steps:
step S1, establishing a hadoop security vulnerability database, and adding a hadoop component as a scanning object;
step S2, generating a scanning task after adding a scanning strategy and configuration to a scanning object;
step S3, calling a scanning engine to execute the scanning task created in the step S2;
step S4, the scanning engine automatically crawls the directory and the file of the scanned object and simultaneously realizes the detection of the scanned object;
step S5, when the scanning task and the scanning strategy in the step S4 do not need to be modified, the scanning task is executed until the end; when the scanning task and the scanning strategy need to be modified, the scanning task is stopped to modify the scanning task and the scanning strategy, and then the modified scanning task is executed again:
and step S6, generating a scanning result after the scanning task is finished, generating a scanning result report based on a preset scanning report template, and providing the scanning result report for a user.
In an embodiment of the present invention, in step S1, the scanned object at least includes: scanning an open port, fingerprint information, information leakage and detection of hadoop version information of the target host, and checking hadoop loopholes, hadoop component safety configuration, detection virtualization loopholes of the hadoop component safety configuration and weak password resource isolation.
In an embodiment of the present invention, in the step S3, the scan engine includes a scan engine for scanning hadoop vulnerabilities, virtualization vulnerabilities, hadoop component security configuration checking, port scanning, target host fingerprint information, and weak password resource isolation checking these objects.
In an embodiment of the present invention, in the step S3, the specific implementation process of the scan engine automatically crawling the directory and the file of the scanned object and simultaneously implementing the detection of the scanned object includes: acquiring vulnerabilities of hadoop components including hbase, hive, MapReduce and HDFS by using a crawler, carrying out vulnerability detection on the hadoop components according to a configurable scanning strategy and scanning configuration, and importing mirror image files in raw, cow, qcow and vmdk formats for detection; the security vulnerability detection, the resource interference detection, the anti-loading malicious software detection, the storage isolation detection, the memory isolation detection, the network isolation detection and the disk user data residue detection of the virtual machine platform with the version of more than KVM1.2, Xen4.0 and Vmware5.0 are realized, and the detection virtualization vulnerability, the weak password resource isolation and the target host fingerprint information including the hadoop vulnerability, the hadoop component security configuration and the hadoop component security configuration in the scanning task are scanned one by calling the corresponding scanning engine according to the scanning strategy set, so that the target vulnerability detection task is completed.
In an embodiment of the present invention, in the step S6, the scan result report includes descriptions of the name, discovery time, corresponding version, bug description, and repair suggestion of the Hadoop security vulnerability, and stores these descriptions in the Hadoop security vulnerability database.
In an embodiment of the invention, in the step S3, the scan engine is a VMM vulnerability checking tool.
Compared with the prior art, the invention has the following beneficial effects: the invention can monitor the system in real time, scan the core components of the system and generate the report at the first time, thereby reducing the damage of the vulnerability to the system and ensuring the security of enterprise data.
Detailed Description
The following specifically describes the technical means of the present invention.
The invention provides a method for detecting hadoop security vulnerabilities, which comprises the following steps:
step S1, establishing a hadoop security vulnerability database, and adding a hadoop component as a scanning object;
step S2, generating a scanning task after adding a scanning strategy and configuration to a scanning object;
step S3, calling a scanning engine to execute the scanning task created in the step S2;
step S4, the scanning engine automatically crawls the directory and the file of the scanned object and simultaneously realizes the detection of the scanned object;
step S5, when the scanning task and the scanning strategy in the step S4 do not need to be modified, the scanning task is executed until the end; when the scanning task and the scanning strategy need to be modified, the scanning task is stopped to modify the scanning task and the scanning strategy, and then the modified scanning task is executed again:
and step S6, generating a scanning result after the scanning task is finished, generating a scanning result report based on a preset scanning report template, and providing the scanning result report for a user.
In step S1, the scanned object at least includes: scanning an open port, fingerprint information, information leakage and detection of hadoop version information of the target host, and checking hadoop loopholes, hadoop component safety configuration, detection virtualization loopholes of the hadoop component safety configuration and weak password resource isolation.
In the step S3, in the step S3, the scan engine includes a scan engine for scanning hadoop vulnerabilities, virtualization vulnerabilities, hadoop component security configuration check, port scan, target host fingerprint information, and weak password resource isolation check of these objects.
In step S3, the specific implementation process of the scan engine automatically crawling the directory and the file of the scanned object and simultaneously implementing the detection of the scanned object is as follows: acquiring vulnerabilities of hadoop components including hbase, hive, MapReduce and HDFS by using a crawler, carrying out vulnerability detection on the hadoop components according to a configurable scanning strategy and scanning configuration, and importing mirror image files in raw, cow, qcow and vmdk formats for detection; the security vulnerability detection, the resource interference detection, the anti-loading malicious software detection, the storage isolation detection, the memory isolation detection, the network isolation detection and the disk user data residue detection of the virtual machine platform with the version of more than KVM1.2, Xen4.0 and Vmware5.0 adopted by the hadoop platform are realized, and the detection virtualization vulnerability including the hadoop vulnerability, the security configuration of the hadoop component, the weak password resource isolation and the target host fingerprint information in the scanning task are scanned one by calling the corresponding scanning engine according to the scanning strategy set, so that the target vulnerability detection task is completed.
In step S6, the scan result report includes descriptions of the name, discovery time, corresponding version, bug description, and repair suggestion of the Hadoop security vulnerability, and these descriptions are stored in the Hadoop security vulnerability database.
The following are specific implementation examples of the present invention.
A detection method for Hadoop security vulnerabilities is sequentially carried out according to the following steps:
the method comprises the following steps: setting a Hadoop security vulnerability database, adding a Hadoop assembly as a scanning object, and realizing the scanning of the object;
step two: generating a scanning task by adding a scanning object, a scanning strategy and configuration in the step one, and realizing the scanning of the scanning object;
step three: calling a scanning engine to execute the scanning task established in the step two;
step four: the called scanning engine in the third step can automatically crawl the directory and the file of the scanned object, and realize the scanning and the detection of the object;
step five: when there is no problem in the scanning task in step four, the scanning task is executed until the end. When the scanning task and the scanning strategy need to be modified, the scanning task is stopped to modify the scanning task and the scanning strategy, and then the modified scanning task is executed again:
step six: and after the scanning task is finished, generating a scanning result report based on a preset scanning report template, and providing the scanning result report for a user.
In this embodiment, the object scanned in the first step at least includes: scanning an open port, fingerprint information, information leakage and hadoop version information detection of a target host, and checking hadoop vulnerability, hadoop component security configuration, detection virtualization vulnerability of hadoop component security configuration and weak password resource isolation.
In this embodiment, the scanning engine in step four crawls and detects the directory and the file of the scanned object, where a crawler is used to obtain vulnerabilities of mainstream hadoop components such as hbase, hive, MapReduce, and HDFS, vulnerability detection is performed on the hadoop components according to a configurable scanning strategy and scanning configuration, and image files in formats such as raw, cow, qcow, and vmdk are imported for detection. The security vulnerability detection of a virtual machine platform with versions more than KVM1.2, Xen4.0 and Vmware5.0 adopted by a hadoop platform, resource interference detection, anti-loading malicious software detection, storage isolation detection, memory isolation detection, network isolation detection and disk user data residue detection are realized, and the hadoop vulnerability, security configuration, virtualization vulnerability, weak password, fingerprint information and the like in a scanning task are scanned one by one according to a scanning strategy set by calling a corresponding scanning engine, so that a target vulnerability detection task is completed.
In this embodiment, the scan report in the sixth step includes descriptions of the name, discovery time, corresponding version, bug description, and repair suggestion of the Hadoop security vulnerability, and stores the descriptions into the Hadoop security vulnerability database in the first step.
In this embodiment, the scan engine in step three is a VMM vulnerability checking tool.
Finally, it should be noted that: although the present invention has been described in detail with reference to the above embodiments, it should be understood by those skilled in the art that: modifications and equivalents may be made thereto without departing from the spirit and scope of the invention and it is intended to cover in the claims the invention as defined in the appended claims.
Claims (3)
1. A method for detecting hadoop security vulnerabilities is characterized by comprising the following steps:
step S1, setting a hadoop security vulnerability database, and adding a hadoop component as a scanning object;
step S2, generating a scanning task after adding a scanning strategy and configuration to a scanning object;
step S3, calling a scanning engine to execute the scanning task created in the step S2;
step S4, the scanning engine automatically crawls the directory and the file of the scanned object and simultaneously realizes the detection of the scanned object;
step S5, when the scanning task and the scanning strategy in the step S4 do not need to be modified, the scanning task is executed until the end; when the scanning task and the scanning strategy need to be modified, the scanning task is stopped to modify the scanning task and the scanning strategy, and then the modified scanning task is executed again:
step S6, generating a scanning result after the scanning task is finished, generating a scanning result report based on a preset scanning report template, and providing the scanning result report for a user;
in step S1, the scanned object at least includes: scanning an open port, fingerprint information, information leakage and hadoop version information detection of a target host, and checking hadoop vulnerability, hadoop component security configuration, virtualization vulnerability of the hadoop component security configuration and weak password resource isolation;
in step S3, the scan engine includes a scan engine for scanning hadoop vulnerabilities, virtualization vulnerabilities, hadoop component security configurations, ports, target host fingerprint information, and weak password resources to isolate these objects;
in step S3, the specific implementation process of the scan engine automatically crawling the directory and the file of the scanned object and simultaneously implementing the detection of the scanned object is as follows: acquiring vulnerabilities of hadoop components including hbase, hive, MapReduce and HDFS by using a crawler, carrying out vulnerability detection on the hadoop components according to a configurable scanning strategy and scanning configuration, and importing mirror image files in raw, cow, qcow and vmdk formats for detection; the security vulnerability detection, the resource interference detection, the anti-loading malicious software detection, the storage isolation detection, the memory isolation detection, the network isolation detection and the magnetic disk user data residue detection of a virtual machine platform adopted by a hadoop platform are realized, and the security vulnerability detection task of a target is completed by calling corresponding scanning engines and scanning virtualization vulnerabilities, weak password resource isolation and target host fingerprint information including hadoop vulnerabilities, hadoop component security configuration and hadoop component security configuration in scanning tasks one by one according to scanning strategies.
2. The method for detecting the Hadoop security vulnerability of claim 1, wherein in the step S6, the scan result report includes descriptions of the name, discovery time, corresponding version, vulnerability description, and repair suggestion of the Hadoop security vulnerability, and stores the descriptions in the Hadoop security vulnerability database.
3. The method for detecting hadoop security vulnerability of claim 1, wherein in step S3, the scan engine is a VMM vulnerability checking tool.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811040881.4A CN109190381B (en) | 2018-09-07 | 2018-09-07 | Method for detecting hadoop security vulnerability |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201811040881.4A CN109190381B (en) | 2018-09-07 | 2018-09-07 | Method for detecting hadoop security vulnerability |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN109190381A CN109190381A (en) | 2019-01-11 |
| CN109190381B true CN109190381B (en) | 2022-05-10 |
Family
ID=64915103
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201811040881.4A Active CN109190381B (en) | 2018-09-07 | 2018-09-07 | Method for detecting hadoop security vulnerability |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN109190381B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110377518B (en) * | 2019-07-17 | 2023-07-25 | 招商银行股份有限公司 | Full-flow scanning method, device, equipment and readable storage medium |
Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104144142A (en) * | 2013-05-07 | 2014-11-12 | 阿里巴巴集团控股有限公司 | Web vulnerability discovery method and system |
-
2018
- 2018-09-07 CN CN201811040881.4A patent/CN109190381B/en active Active
Patent Citations (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104144142A (en) * | 2013-05-07 | 2014-11-12 | 阿里巴巴集团控股有限公司 | Web vulnerability discovery method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN109190381A (en) | 2019-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8813235B2 (en) | Expert system for detecting software security threats | |
| US10083277B2 (en) | Secure computing systems and methods | |
| Vu et al. | Lastpymile: identifying the discrepancy between sources and packages | |
| CN106446707A (en) | Dynamic data leakage prevention system and method | |
| US9703974B1 (en) | Coordinated file system security via rules | |
| CN109997143A (en) | The safety of sensitive data is shared | |
| CN103366122B (en) | For realizing the method and system of scan service | |
| CN113138836B (en) | Escape prevention method using escape prevention system based on Docker container | |
| CN106203108A (en) | Linux white list system protection method based on kernel module and device | |
| CN114422197A (en) | Permission access control method and system based on policy management | |
| CN109190381B (en) | Method for detecting hadoop security vulnerability | |
| CN106326733A (en) | Method and apparatus for managing applications in mobile terminal | |
| CN106203105B (en) | File management method and device | |
| Gu et al. | Continuous intrusion: Characterizing the security of continuous integration services | |
| WO2017167015A1 (en) | Method and device for server device security management and computer storage medium | |
| CN112148709A (en) | Data migration method, system and storage medium | |
| US11151274B2 (en) | Enhanced computer objects security | |
| CN116680699A (en) | Vulnerability priority ordering system, vulnerability priority ordering method, computer equipment and storage medium | |
| CN102868690B (en) | Method and system for WEB service isolation and detection | |
| US20200195651A1 (en) | Account lifecycle management | |
| Pusuluri | Taxonomy Of Security and Privacy Issues in Serverless Computing | |
| CN106372463A (en) | Middleware protection method, apparatus and system | |
| CN107193963B (en) | Distributed development method of database application system | |
| Haar et al. | Securing orchestrated containers with bsi module sys. 1.6 | |
| US20240348664A1 (en) | Security policy analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |