CN115964713A - Safety evaluation method and system for self-development information system codes in large-scale enterprise - Google Patents

Abstract

The invention provides a safety evaluation method and a safety evaluation system for self-development information system codes in a large enterprise. The invention realizes the safety detection and high-risk vulnerability scanning of the codes of the self-development system in the enterprise by utilizing a code safety visual integrated management technology, a distributed message queue technology, a flow acquisition technology and a vulnerability detection technology, effectively solves the problem that the safety vulnerabilities of the self-development information system are frequent after being online, and provides real-time and reliable data support for the code safety situation and the information safety situation of each level of development departments for a company informatization management layer through the visual integrated management technology. In addition, the method is suitable for a development team of the self-owned information system in a large enterprise, and by moving the security testing process to the development stage, the information security vulnerability detection is carried out on the code side before the information system is on line, so that the information security vulnerability can be timely and effectively discovered.

Description

Safety evaluation method and system for self-development information system codes in large-scale enterprise

Technical Field

The invention relates to the technical field of safety of self-development information system codes, in particular to a safety evaluation method and a safety evaluation system of self-development information system codes in a large enterprise.

Background

While technology subversive trends such as cloud computing and big data continue to play a role in application economy, requirements such as rapid business iteration are indispensable in business decision making. Under the large environment of application driving, cloud computing and mobility, the value of the assisted service is added through rapid service iteration.

Given the need for rapid business iterations, many of the links in traditional security procedures have become obstacles that must be eliminated, unfortunately many enterprises are unaware of these problems. The traditional safe operation is based on that after the system is released, the safety personnel confirm the safety risk of the system. The flow designed in this way is only suitable for the business activities in waterfall mode. Unfortunately, with the introduction of iterations, such a secure manner of operation is flawed and poses inherent risks within the system, as business decisions need to balance the inlining and keep up with the speed of business.

With the rapid change of business decisions, traditional security is no longer an option. During the development cycle, it is too late and not fast enough to cooperate with iterative design and system release, and traditional security is in a more embarrassing position with the need for rapid business iterations.

Generally, a security team cannot collect all the information needed to make meaningful security decisions. The value creation process continues to accelerate in order to provide iterative values that closely map customer requirements. So that a decision at the end of a cycle or a complete system test may lead to fatal results. In fact, most of these security decisions are rarely adopted and are often rejected by the business executive, who can first be questioned once a security incident occurs.

Therefore, the safety problem can be found before the system is released, and even in the testing and developing stage, the resonance which becomes the safety personnel can be timely repaired. However, the current mainstream security products either cannot automatically, quickly and comprehensively discover the vulnerability or cannot visually present the vulnerability in the hands of developers; along with service iteration, security personnel are required to continuously initiate scanning tasks, scanning results of each time are manually collated into scanning reports which are easy to understand and are sent to the hands of related personnel, and the security personnel perform rechecking after repair, so that the working efficiency of the security personnel is greatly reduced, and the requirement of rapid service iteration cannot be met. How to perform security detection on a new service system before online becomes a problem of important attention of each enterprise.

The Chinese patent application with publication number CN104767757A, published as 2015, 07, 08 and 2015, discloses a multi-dimensional security monitoring method and system based on WEB services, wherein website security is comprehensively monitored from three dimensions of availability, security events and Web vulnerabilities, a monitoring and scanning module performs data detection on a target website by using a port detection engine, and then scanning information is transmitted to a data center. The data center effectively evaluates the safety condition of the website through correlation analysis of information, and classifies the website into a high-risk website, a medium-risk website, a low-risk website and a safety website. The invention realizes the comprehensive monitoring of the website safety from a plurality of dimensions, has higher efficiency, can dynamically display the monitoring result and realizes the safety monitoring of each service node in the scanned network. However, the inventive patent application does not relate to pre-release security testing of self-developed information system code within large enterprises.

The Chinese patent application with the publication number of CN104809404A and the publication date of 07, 29 and 2015 discloses a data layer system of an information security attack and defense platform, which is provided with a tool library module, a scene configuration library module, a courseware library module, a security information library module, a log library module, an attack behavior library module and a platform library module, wherein a complete, efficient and uniform data layer system is provided for the information security attack and defense platform in a mode of combining virtual equipment and physical equipment through independent operation of seven resource library modules, linkage among the modules and linkage of the modules and external data, so that the information security attack and defense platform is helped to effectively realize network security evaluation, attack and defense confrontation, new product inspection and test, and a series of evaluation and evaluation on the aspects of network architecture, design process, host security, data security and the like. However, the inventive patent application is not directed to pre-release security testing of self-developed information system code within large enterprises.

Disclosure of Invention

The present invention aims to address at least one of the above-mentioned deficiencies of the prior art. For example, one of the purposes of the invention is to solve the problem that information security holes are frequently generated due to the fact that the self-development information system in an enterprise has high iteration speed in a development cycle and the security testing process intervenes later and the personnel of the security team in the enterprise are insufficient.

In order to achieve the above object, an aspect of the present invention provides a security assessment method for self-development information system codes inside a large enterprise, including the following steps:

step one, when the conventional function test is carried out on the information system, a flow acquisition module automatically captures http/https flow in a completely transparent mode for a user, and all the finally acquired flow is stored in a storage center database;

analyzing the request flow and the data flow in the storage center database through a flow analysis module, analyzing a request mode, an interface URL and a request body of a target site, and finally, structurally storing the request mode, the interface URL and the request body into the storage center database for a subsequent module to call;

step three, calling a crawler module, trying to initiate an HTTP request to a target address according to the domain name or the address of the target website extracted from the distributed message queue to perform HTML text crawling, then analyzing URI endpoints in the responded HTML text, and simultaneously storing the request of the crawler, the responded HTML text and the URI endpoints into a storage center database for calling of a subsequent module;

step four, the vulnerability detection engine takes out the identified information system URL from the storage center database, performs multi-thread scanning on all the URLs of the information system, and inserts the scanning result into the visual integrated management platform database;

and fifthly, analyzing code security vulnerability scanning result data stored in the storage center database, and performing vulnerability analysis and chart display.

Another aspect of the present invention provides a security assessment system for self-development of information system codes in a large enterprise, the security assessment system comprising a traffic collection module, a traffic analysis module, a crawler module, a vulnerability detection engine, a storage center database, and a visual integrated management platform database, wherein,

the flow acquisition module can automatically capture the flow of http/https in a completely transparent manner for a user when the information system is subjected to conventional function test, and finally all the acquired flow is stored in a storage center database;

the flow analysis module can analyze the request flow and the data flow in the storage center database, analyze the request mode of a target site, an interface URL and a request body, and finally structurally store the request mode, the interface URL and the request body in the storage center database for being called by a subsequent module;

the crawler module can be called to realize that an HTTP request is tried to be initiated to a target address to perform HTML text crawling according to a domain name or an address of a target website extracted from a distributed message queue, then URI endpoints in a response HTML text are analyzed, and the crawler request, the response HTML text and the URI endpoints are stored in a storage center database for being called by a subsequent module;

the vulnerability detection engine can take out the identified information system URL from the storage center database, perform multi-thread scanning on all the URLs of the information system, and insert the scanning result into the visual integrated management platform database;

and the visual integrated management platform database can perform vulnerability analysis and chart display on the scanning result.

Compared with the prior art, the invention has the beneficial effects of at least one of the following contents:

by moving the security test flow to the development stage, information security loopholes can be detected on the code side before the information system is on line, the information security loopholes can be found timely and effectively, and the problem that the information security loopholes are frequently generated due to the fact that the self-development information system in an enterprise is high in iteration speed in the development period and the security test flow intervenes later is solved;

the code security visualization integrated management technology, the distributed message queue technology, the flow acquisition technology and the vulnerability detection technology can be used for realizing the security detection and high-risk vulnerability scanning of the self-development system codes in the enterprise, and the problem that the self-development information system has frequent security vulnerabilities after being online is effectively solved;

the method can provide real-time and reliable data support for code safety conditions and information safety conditions of development departments at all levels for an information management layer of a company through a visual integrated management technology.

Drawings

The above and other objects and/or features of the present invention will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

FIG. 1 shows a schematic flow diagram of an exemplary embodiment of the present invention;

FIG. 2 illustrates a GET request connection diagram in an exemplary embodiment of the invention;

FIG. 3 illustrates a POST request connection in an exemplary embodiment of the invention.

Detailed Description

Hereinafter, the security evaluation method and system for self-development information system code inside a large enterprise according to the present invention will be described in detail with reference to exemplary embodiments. The self-development information system in the large enterprise is supported by an information technology and a communication technology, has a huge scale and wide distribution, adopts a multi-level network structure, spans a plurality of security domains, processes massive, complex and diverse data, and provides a large system for various types of applications.

Generally, the safety evaluation method and the safety evaluation system for the self-development information system codes in the large-scale enterprise can be suitable for a self-owned information system development team in the large-scale enterprise, and information safety loopholes are detected on the code side before the information system is on line by moving the safety test process to the development stage, so that the information safety loopholes can be found timely and effectively. The invention realizes the safety detection and high-risk vulnerability scanning of the codes of the self-development system in the enterprise by utilizing a code safety visual integrated management technology, a distributed message queue technology, a flow acquisition technology and a vulnerability detection technology, effectively solves the problem that the safety vulnerabilities of the self-development information system are frequent after being online, and provides real-time and reliable data support for the code safety situation and the information safety situation of each level of development departments for a company informatization management layer through the visual integrated management technology.

FIG. 1 shows a schematic flow diagram of an exemplary embodiment of the present invention; fig. 2 illustrates a GET request connection diagram in an exemplary embodiment of the invention, where a GET request: request data from a specified resource, SYN: synchronization sequence number, ACK: confirmation character, GET Head: a response message header requesting to acquire resources; FIG. 3 illustrates a POST request connection diagram in an exemplary embodiment of the invention, wherein the POST request: submitting the data to be processed to the specified resource, SYN: synchronization sequence number, ACK: confirmation character, DATA: and (6) data.

As shown in fig. 1 to 3, in an exemplary embodiment of the present invention, a security evaluation method for self-developed information system code inside a large enterprise is implemented by the following steps:

firstly, collecting http/https flow, automatically capturing the flow by a collecting module when testers in an enterprise perform conventional function tests on an information system for safety testing, completely transparent to users, and finally storing all collected flow to a storage center database.

The flow acquisition module comprises an agent module, a pile inserting module, a flow sniffing module and a log importing module.

1) The proxy module is a clustered functional module utilizing gateway proxy technology. Identity authentication is carried out, service flow is copied to a storage center, and effective load balancing can be achieved under the high concurrency condition generated when service access amount is centralized in a clustered proxy mode. The self-scheduling cluster agent processes the request from the client and forwards the request to the storage center. According to the invention: proxy clustering can reduce the number of single points of failure and achieve high availability of clustered resources. The method can effectively solve the problems of overlarge data flow and overweight network load, does not need to spend expensive expenses to purchase the server with excellent performance, fully utilizes the existing equipment, and avoids the loss of the data flow caused by single-point failure of the server. The method has flexible and various balancing strategies to reasonably distribute data traffic to the common burden of the servers in the server group. Even if the existing server is expanded and upgraded, a new server is simply added to the service group without changing the existing network structure and stopping the existing service.

2) The pile inserting module is used for linkage detection of a passive safety testing technology and an active safety testing technology. According to the invention: the passive security test needs to deploy an Agent probe in a Web Server, inject tracking codes into a bytecode program by using an instrumentation technology to form instrumented bytecode, and directly acquire data streams and collect related information when the program runs. The instrumentation mode can acquire information such as code data stream, HTTP request/response packet and the like of the tested item only by one-key starting on the tested service server, and sends the request and the data stream to the storage center database.

3) The traffic sniffing module is mainly deployed on a service server through a traffic collection Agent under the condition that test environments of a PC (personal computer) end and a mobile end are complex, and HTTP traffic is obtained in a sniffing mode and is sent to a storage center database. The invention has no perception on the tester, removes extra configuration work and does not change the working process of the tester. Meanwhile, the flow collection Agent acquires the HTTP flow in a sniffing mode, and the HTTP flow is in a bypass relation with a service data link, so that the flow direction of service data is not influenced at all.

4) The log import module is mainly used for recording complete WEB logs by aiming at a log platform which is built by an enterprise, and has a complete log import interface. According to the invention: the enterprise imports the logs in the log platform into the distributed message queue system according to the agreed format, the log analysis module extracts elements such as a Header, a Cookie, a User-Agent and a Body in the logs, the logs are recombined based on the HTTP protocol, the original request is restored, and the restored request is sent to the storage center database. The maximum coverage of the interface can be achieved and the detection effect can be optimized by converting abundant access logs, function test logs, user behavior logs and the like recorded in the log platform into original requests to be stored and delivered to the vulnerability detection engine for detection.

Each acquisition module corresponds to different service scenes, and the use modes are different, wherein the pile insertion module can support various risk detections such as general bugs, service logic bugs, third-party components and the like, and can cover disposable interface scenes such as encryption, verification codes and the like, so that the service scenes are completely covered; the instrumentation module does not need to replay requests and dirty data, almost achieves 0 false alarm, improves the detection precision, can directly position code positions, code contents and data stream information, completely shows the whole process of vulnerability input and propagation to final execution, and greatly reduces vulnerability reproduction difficulty.

And step two, analyzing the request flow and the data flow in the database of the storage center through a flow analysis module.

The flow analysis module is mainly used for analyzing HTTP protocol flow, collecting normal request and response data in normal functional test, and analyzing HTTP request protocol packets, including methods, protocol versions and resource URIs in a request line, protocol fields and field values in a request head, and parameters and parameter values in a request body. And analyzing the HTTP response protocol packet, including the protocol version and the status code request flow returned in the response line, the parameters and the parameter values in the response header and the HTML text in the response body.

And step three, calling a crawler module, trying to initiate an HTTP request to a target address to perform HTML text crawling according to the domain name or address of the target website extracted from the distributed message queue, then analyzing URI endpoints in the responded HTML text, and simultaneously storing the request of the crawler, the responded HTML text and the URI endpoints in a storage center database for calling of subsequent modules.

The crawler module has the following working process:

1) The crawler module acquires a URL (uniform resource locator) address character string of the information system from the distributed message queue system, segments the URL address character string through regular matching, and then obtains an independent protocol type character string (http/https), an address (IP (Internet protocol) or a domain name) and parameter information. This completes the identification and segmentation of the URL address.

2) A Request object is constructed by adding a Header Request head and Request parameters, then a standard HTTP method call (GET) is initiated to the segmented target address, and the target server responds to a standard HTML text, thus completing the step of acquiring the HTML text of the target site.

3) And converting the HTML text into a DOM tree, acquiring corresponding tag content, attributes and attribute values through corresponding HTML tags, extracting URI endpoints related to the target address through matching and screening, and storing the URI endpoints into a storage center database, so that the steps of analyzing the HTML text and extracting the URI endpoints are completed.

And step four, the vulnerability detection engine takes out the identified information system URL from the storage center database, performs multi-thread scanning on all the URLs of the information system, and inserts the scanning result into the visual integrated management platform database.

The vulnerability detection engine rapidly detects the item vulnerability through a cluster detection method. The cluster (cluster) technology can obtain relatively high benefits in the aspects of performance, reliability and flexibility with low cost, and the task scheduling is the core technology in the cluster system. Meanwhile, the detection cluster simulates various vulnerability detection modes to carry out comprehensive vulnerability detection on the requests in the user configuration project by acquiring the request flow data of the storage center, and quickly generates a vulnerability report; and automatically detecting along with the change of the service request, and updating the newly added detection result into the bug report.

Analyzing code security vulnerability scanning result data stored in a storage center database, counting the vulnerability number, ranking the project vulnerabilities, distributing vulnerability trends, displaying charts through a code security visualization integrated management console, and providing the charts to an information management layer to serve as decision support.

After the security evaluation method of the above exemplary embodiment is defined, an organization development tester and a professional code security audit team use the security evaluation method of the exemplary embodiment to perform full code audit on the following systems respectively, and a summary report of related results is as follows:

example 1 Integrated platform summary report

1. Source code information

2. Rank statistics

3. Classification statistics

/>

/>

Example 2 contractor management System summary report one, source code information

2. Rank statistics

3. Classification statistics

/>

/>

Example 3 scientific System summary report

1. Source code information

2. Rank statistics

/>

3. Classification statistics

/>

In another exemplary embodiment of the invention, the security evaluation system for self-development of information system codes in large enterprises is composed of a traffic collection module, a traffic analysis module, a crawler module, a vulnerability detection engine, a storage center database and a visual integrated management platform database. The system can realize the safety evaluation method for self-development information system codes in the large-scale enterprise in the exemplary embodiment.

The flow acquisition module can automatically capture http/https flow in a completely transparent manner for a user when performing conventional function testing on an information system, and all the finally acquired flow is stored in a storage center database. The flow acquisition module comprises an agent module, a plug-in module, a flow sniffing module and a log import module, wherein the plug-in module can support the risk detection of general bugs, business logic bugs and third-party components and can cover the one-time interface scenes of encryption and verification codes; the instrumentation module can directly position to a code position, code content and data stream information without replaying requests and dirty data, and completely show the whole process of vulnerability from input and propagation to final execution.

The flow analysis module can analyze the request flow and the data flow in the storage center database, analyze the request mode, the interface URL and the request body of the target site, and finally structurally store the request mode, the interface URL and the request body in the storage center database so as to be called by a follow-up module. The traffic analysis module can analyze HTTP protocol traffic, collect request and response data in functional test, and analyze HTTP request protocol packet and HTTP response protocol packet, wherein the HTTP response protocol includes method, protocol version and resource URI in request line, protocol field and field value in request head, and parameters and parameter values in request body; the HTTP response protocol comprises a protocol version and a status code returned in a response line, parameters and parameter values in a response header and HTML text in a response body.

The crawler module can be called to realize that an HTTP request is tried to be initiated to a target address to perform HTML text crawling according to a domain name or an address of a target website extracted from a distributed message queue, then URI endpoints in a response HTML text are analyzed, and the crawler request, the response HTML text and the URI endpoints are stored in a storage center database for subsequent module calling.

The working process of the crawler module is as follows:

1) The crawler module acquires a URL (uniform resource locator) address character string of an information system from a distributed message queue system, segments the URL address character string through regular matching, and then acquires an independent protocol type character string and address and parameter information to complete the identification and segmentation of a URL address;

2) A Request object is constructed by adding a Header Request head and Request parameters, then a standard HTTP method call is initiated to the segmented target address, and a target server responds to a standard HTML text, thereby completing the step of obtaining the HTML text of the target site;

3) And converting the HTML text into a DOM tree, acquiring corresponding tag content, attributes and attribute values through corresponding HTML tags, extracting URI endpoints related to the target address through matching and screening, and storing the URI endpoints into a storage center database, thereby completing the steps of analyzing the HTML text and extracting the URI endpoints.

The vulnerability detection engine can take out the identified information system URL from the storage center database, perform multi-thread scanning on all the URLs of the information system, and insert the scanning result into the visual integrated management platform database.

And the visual integrated management platform database can perform vulnerability analysis and chart display on the scanning result.

In summary, the invention is a mode method for self-development of information system code security evaluation in a large enterprise, which is researched by the inventor when the inventor performs security detection before a new business system is online. By adopting the method, development testers can rapidly position problems occurring in code compiling in a development test domain through a visual integrated management technology, immediately adjust code content deconstruction, and ensure the safety and reliability of the whole code structure; the company informatization management layer can provide real-time and reliable data support for the code security condition and the information security condition of development departments at all levels.

Although the present invention has been described above in connection with the exemplary embodiments and the accompanying drawings, it will be apparent to those of ordinary skill in the art that various modifications may be made to the above-described embodiments without departing from the spirit and scope of the claims.

Claims (13)

1. A safety evaluation method for self-development information system codes in a large-scale enterprise comprises the following steps:

step one, when the conventional function test is carried out on the information system, a flow acquisition module automatically captures http/https flow in a completely transparent mode for a user, and all the finally acquired flow is stored in a storage center database;

analyzing the request flow and the data flow in the storage center database through a flow analysis module, analyzing a request mode, an interface URL and a request body of a target site, and finally, structurally storing the request mode, the interface URL and the request body into the storage center database for a subsequent module to call;

step three, calling a crawler module, trying to initiate an HTTP request to a target address according to the domain name or the address of the target website extracted from the distributed message queue to perform HTML text crawling, then analyzing URI endpoints in the responded HTML text, and simultaneously storing the request of the crawler, the responded HTML text and the URI endpoints into a storage center database for calling of a subsequent module;

step four, the vulnerability detection engine takes out the identified URL of the information system from the storage center database, carries out multi-thread scanning on all the URLs of the information system and inserts the scanning result into the visual integrated management platform database;

and step five, analyzing code security vulnerability scanning result data stored in the storage center database, and performing vulnerability analysis and chart display.

2. The method according to claim 1, wherein the traffic collection module comprises an agent module, a instrumentation module, a traffic sniffing module, and a log import module.

3. The safety evaluation method for the self-development information system codes in the large-scale enterprise according to claim 2, wherein the instrumentation module can support risk detection of general bugs, business logic bugs and third-party components, and can cover a one-time interface scene of encryption and verification codes; the instrumentation module can directly position code positions, code contents and data stream information without replaying requests and dirty data, and completely show the whole process of vulnerability from input and propagation to final execution.

4. The method as claimed in claim 1, wherein the traffic analysis module is capable of analyzing HTTP protocol traffic, collecting request and response data, and parsing HTTP request protocol packets and HTTP response protocol packets during functional testing.

5. The method as claimed in claim 4, wherein the HTTP response protocol includes method, protocol version and resource URI in request line, protocol field and field value in request header, and parameter value in request body.

6. The method as claimed in claim 4, wherein the HTTP response protocol comprises the protocol version and status code returned in the response line, the parameters and parameter values in the response header, and the HTML text in the response body.

7. The safety evaluation method for self-development information system codes in large-scale enterprises according to claim 1, wherein the workflow of the crawler module is as follows:

1) The crawler module acquires a URL (uniform resource locator) address character string of an information system from a distributed message queue system, segments the URL address character string through regular matching, and then acquires an independent protocol type character string and address and parameter information to complete the identification and segmentation of a URL address;

2) A Request object is constructed by adding a Header Request head and Request parameters, then a standard HTTP method call is initiated to the segmented target address, and a target server responds to a standard HTML text, thereby completing the step of obtaining the HTML text of the target site;

3) And converting the HTML text into a DOM tree, acquiring corresponding tag content, attributes and attribute values through corresponding HTML tags, extracting URI endpoints related to the target address through matching and screening, and storing the URI endpoints into a storage center database, thereby completing the steps of analyzing the HTML text and extracting the URI endpoints.

8. The safety evaluation method for self-development information system codes inside a large enterprise according to claim 1, wherein the vulnerability detection engine rapidly detects a project vulnerability through a method of detecting a cluster.

9. The method according to claim 1, wherein the vulnerability analysis comprises vulnerability number statistics, project vulnerability ranking and vulnerability trend distribution, and the chart display is performed through a code security visualization integrated management console.

10. The safety evaluation system for self-development of information system codes in large enterprises is characterized by comprising a flow acquisition module, a flow analysis module, a crawler module, a vulnerability detection engine, a storage center database and a visual integrated management platform database, wherein the flow acquisition module, the flow analysis module, the crawler module, the vulnerability detection engine and the storage center database are connected in series,

the flow acquisition module can automatically capture http/https flow in a completely transparent mode for a user when a conventional function test is performed on an information system, and all the finally acquired flow is stored in a storage center database;

the flow analysis module can analyze the request flow and the data flow in the storage center database, analyze the request mode of a target site, an interface URL and a request body, and finally structurally store the request mode, the interface URL and the request body in the storage center database for being called by a subsequent module;

the crawler module can be called to realize that an HTTP request is tried to be initiated to a target address to perform HTML text crawling according to a domain name or an address of a target website extracted from a distributed message queue, then URI endpoints in a response HTML text are analyzed, and the crawler request, the response HTML text and the URI endpoints are stored in a storage center database for being called by a subsequent module;

the vulnerability detection engine can take out the identified information system URL from the storage center database, perform multi-thread scanning on all the URLs of the information system, and insert the scanning result into the visual integrated management platform database;

and the visual integrated management platform database can perform vulnerability analysis and chart display on the scanning result.

11. The system for safety assessment of self-development information system codes inside large enterprises according to claim 10, wherein the traffic collection module comprises an agent module, a instrumentation module, a traffic sniffing module, and a log import module, and the instrumentation module can support risk detection of general vulnerabilities, business logic vulnerabilities, and third party components, and can cover one-time interface scenarios of encryption and verification codes; the instrumentation module can directly position to a code position, code content and data stream information without replaying requests and dirty data, and completely show the whole process of vulnerability from input and propagation to final execution.

12. The system of claim 10, wherein the traffic analysis module is capable of analyzing HTTP protocol traffic, collecting request and response data, parsing HTTP request protocol packets and HTTP response protocol packets in a functionality test, the HTTP response protocol including methods, protocol versions and resource URIs in a request line, protocol fields and field values in a request header, and parameters and parameter values in a request body; the HTTP response protocol comprises a protocol version and a status code returned in a response line, parameters and parameter values in a response head, and HTML text in a response body.

13. The system of claim 10, wherein the crawler module has the following workflow:

1) The crawler module acquires a URL address character string of an information system from a distributed message queue system, segments the URL address character string through regular matching, and then obtains an independent protocol type character string and address and parameter information to complete the identification and segmentation of a URL address;

2) A Request object is constructed by adding a Header Request head and Request parameters, then a standard HTTP method call is initiated to the segmented target address, and a target server responds to a standard HTML text, thereby completing the step of acquiring the HTML text of the target site;

3) And converting the HTML text into a DOM tree, acquiring corresponding tag content, attributes and attribute values through corresponding HTML tags, extracting URI endpoints related to the target address through matching and screening, and storing the URI endpoints into a storage center database, thereby completing the steps of analyzing the HTML text and extracting the URI endpoints.

Images