CN113065078B - Statistical analysis method for simulating user behavior to dial and test multistage domain names of WEB sites - Google Patents

Abstract

The invention provides a statistical analysis method for simulating user behavior dialing and testing of a multistage domain name of a WEB site, which comprises the following steps: extracting a test website name in the test URL, and referring the local URL table by taking the test website name as a search object; obtaining a test organization corresponding to the test website name from the local URL table, and generating an IP address feature library according to the test organization, wherein the IP address feature library comprises a BGP address table, a static address table and a cloud service address table; analyzing the webpage source code content corresponding to the test URL to obtain a next-level webpage URL; monitoring a REQUEST sent when the browser accesses the URL of the next-level webpage, analyzing a domain name in the REQUEST, matching an A record corresponding to the domain name with an IP feature library, and judging the attribute of the A record; and writing the domain name, the CNAME and A records corresponding to the domain name and the attributes of the A records into an EXCEL table. The statistical analysis method for simulating the user behavior dialing and testing of the multistage domain names of the WEB sites is automatically completed by the server side, and is high in efficiency and accuracy.

Description

Statistical analysis method for simulating user behavior to dial and test multistage domain names of WEB sites

Technical Field

The invention relates to the technical field of internet, in particular to a statistical analysis method for simulating user behavior dialing and testing of a multistage domain name of a WEB site.

Background

A Domain Name (Domain Name), which is called Domain Name or Domain for short, is the Name of a certain computer or computer group on the internet composed of a string of characters separated by dots, and is used for identifying the electronic orientation of the computer during data transmission. The domain name can be said to be a proxy for an IP address, in order to facilitate remembering the latter. The domain name has uniqueness, and when the IP address is changed by the resource, the resource can be moved to different physical positions in the network address topology only by converting a new IP address and a constant domain name. Based on the above two characteristics, the domain name is also used to create a unique identification for the individual. When any organization or person provides internet resources, the domain name corresponding to the name of the organization or person can be selected, and other persons can easily access the resources.

The statistical analysis of the domain name information in the webpage is important for each enterprise and operator, and the enterprise can improve the access quality and user experience of the website by statistically analyzing the domain name information of the website of the enterprise. The operator can conveniently carry out network charging by counting and analyzing the domain name information of the customer website, and the analysis of the resource distribution of the customer website is convenient for service expansion. However, at present, the statistical analysis is manually performed, which not only requires a large amount of manual processing and analysis, but also has low accuracy.

Disclosure of Invention

The technical problem to be solved by the invention is as follows: in order to solve the problems of low efficiency and low accuracy of people brought by manually carrying out statistical analysis on enterprise website resource information in the prior art, the invention provides a statistical analysis method for testing the multistage domain names of the WEB website by dialing by simulating user behaviors to solve the problems.

The technical scheme adopted by the invention for solving the technical problems is as follows: the statistical analysis method for simulating the user behavior to dial and test the multistage domain names of the WEB sites comprises the following steps:

s1, establishing a local URL table, wherein the local URL table comprises at least one organization and a website name corresponding to the organization;

s2, extracting a test website name in the test URL, and referring the local URL table by taking the test website name as a search object;

if the test website name is found in the local URL table, obtaining a test organization corresponding to the test website name from the local URL table, and generating an IP address feature library according to the test organization, wherein the IP address feature library comprises a BGP address table, a static address table and a cloud service address table;

if the test website name is not found in the local URL table, an IP address feature library is generated, wherein the IP address feature library comprises a cloud service address table;

s3, acquiring webpage source code content and a webpage source code encoding mode corresponding to the test URL, analyzing the webpage source code content by using the webpage source code encoding mode to obtain a next-stage webpage URL and storing the next-stage webpage URL into a second-stage URL queue; sequentially carrying out the steps S4-S10 on the next-level webpage URL in the second-level URL queue;

s4, starting a browser listener to monitor a REQUEST sent by a browser;

s5, starting a browser and inputting the URL of the next-level webpage into the browser for access;

s6, a browser monitor monitors a REQUEST sent when the browser accesses a next-level webpage URL, resolves a domain name in the REQUEST, and stores the domain name in a domain name queue after carrying out duplicate removal processing on the domain name;

s7, closing the browser monitor and the browser;

s8, resolving corresponding CNAME and A records for each domain name in the domain name queue;

s9, matching the record A corresponding to the domain name with an IP feature library, and judging the attribute of the record A;

and S10, writing the domain name in the domain name queue, the CNAME and A record corresponding to the domain name and the attribute of the A record into an EXCEL table.

Preferably, in step S9, the record a corresponding to the domain name is sequentially matched with addresses in the BGP address table, the static address table, and the cloud service address table, and when matching is successful, the attribute of the record a is set as the name of the address table where the matching address is located, and if matching is failed, the attribute of the record a is set as interconnection.

Preferably, in step S8, if the CNAME is not resolved, the CNAME is set to null.

Preferably, in step S8, if the number of a records corresponding to the domain name is two or more, all the a records are collected into an a record list corresponding to the domain name.

Preferably, a flag bit corresponding to the test URL is set, and in step S2, if the test site name is found in the local URL table, the flag bit is set to True, and if the test site name is not found in the local URL table, the flag bit is set to False;

in step S2, if the flag bit is True, acquiring a test organization corresponding to the test site name, and acquiring a BGP address table and a static address table corresponding to the test organization and used for generating an IP address feature library; and if the flag bit is False, using a cloud service address table corresponding to the test website name to generate the IP address feature library.

Preferably, the method further comprises the following steps:

before the step S2, the user enters a test interface after logging in, the test URL is obtained on the test interface and whether the test URL is reachable is tested, if yes, the step S2 is carried out, and if not, information indicating unreachability is returned to the user.

Preferably, the method further comprises the following steps:

and returning the EXCEL form to the user terminal.

The method has the advantages that the statistical analysis method for simulating the user behavior dialing and testing of the multistage domain names of the WEB sites is automatically completed by the server, and is high in efficiency and accuracy.

Drawings

The invention is further illustrated with reference to the following figures and examples.

FIG. 1 is a flowchart of an embodiment of a statistical analysis method for simulating user behavior to dial and test multi-level domain names of a WEB site according to the present invention.

Fig. 2 is a simple flowchart of the method for obtaining the IP feature library according to the embodiment of the present invention.

Fig. 3 is an exemplary diagram of a local URL table according to an embodiment of the statistical analysis method for simulating user behavior dialing and testing of the multi-level domain names of the WEB sites of the present invention.

Fig. 4 is a schematic diagram of listening to a REQUEST from a browser.

Fig. 5 is a schematic diagram of an attribute setting process of record a in the embodiment of the statistical analysis method for dialing and testing the multilevel domain name of the WEB site by simulating the user behavior.

Detailed Description

Reference will now be made in detail to embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to the same or similar elements or elements having the same or similar function throughout. The embodiments described below with reference to the accompanying drawings are illustrative only for the purpose of explaining the present invention and are not to be construed as limiting the present invention.

In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "axial", "radial", "circumferential", and the like, indicate orientations and positional relationships based on the orientations and positional relationships shown in the drawings, and are used merely for convenience of description and for simplicity of description, and do not indicate or imply that the device or element being referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore, should not be considered as limiting the present invention.

Furthermore, the terms "first," "second," and the like are used for descriptive purposes only and are not to be construed as indicating or implying relative importance. In the description of the present invention, it should be noted that, unless explicitly stated or limited otherwise, the terms "connected" and "connected" are to be interpreted broadly, e.g., as being fixed or detachable or integrally connected; can be mechanically or electrically connected; may be directly connected or indirectly connected through an intermediate. The specific meanings of the above terms in the present invention can be understood in specific cases to those skilled in the art. In addition, in the description of the present invention, "a plurality" means two or more unless otherwise specified.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and alternate implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the present invention.

As shown in fig. 1 and 2, the present invention provides a statistical analysis method for simulating user behavior to dial and measure a multi-level domain name of a WEB site, which is characterized by comprising the following steps:

a local URL table is established in a server, and the local URL table comprises at least one organization and a website name corresponding to the organization; for example, as shown in fig. 3, the local URL table is organized by Ali, tencent, 360, huawei, etc., the local URL table has website names www.taobao.com, www.youku.com.. Tencent has website names www.qq.com, v.qq.com.

A user logs in at a user side, the user side enters a test interface after logging in, and the user inputs a test URL in the test interface;

the server side acquires a test URL from the client side and requests for testing a URL network, if the return value is 404, the test URL network is not communicated, information indicating that the test URL network is inaccessible is returned to the client side, and if the test URL network is communicated, the next step is carried out;

extracting a test website name in the test URL, and referring to a local URL table by taking the test website name as a search object;

setting a flag bit corresponding to the test URL, setting the flag bit to True if the test website name is found in the local URL table, and setting the flag bit to False if the test website name is not found in the local URL table;

when the flag bit is True, acquiring a test organization corresponding to the test website name, acquiring a BGP address table, a static address table, a cloud service address table and an Education (EDU) address table corresponding to the test organization according to the test organization, and generating an IP address feature library by using the BGP address table, the static address table, the cloud service address table and the Education (EDU) address table; and if the flag bit is False, only a cloud service address table and an Education (EDU) address table are obtained, and the cloud service address table and the Education (EDU) address table corresponding to the test website name are used for generating an IP address feature library.

Acquiring webpage source code content and a webpage source code encoding mode corresponding to the test URL, analyzing the webpage source code content by using the webpage source code encoding mode to obtain a next-stage webpage URL, and storing the next-stage webpage URL into a second-stage URL queue; sequentially carrying out the following steps on the next-level webpage URL in the second-level URL queue:

starting a browser listener to monitor a REQUEST sent by a browser; the principle of the browser listener listening to the REQUEST of the browser is shown in fig. 4, and the process of the user normally using the browser to access the website includes: the URL is input in the browser, the IP corresponding to the domain name is inquired from the DNS, the browser sends a resource request to a WEB server of the IP address after obtaining the corresponding IP address, the WEB server obtains the resource according to the request, and the WEB server returns the resource to the browser. Monitoring when the browser sends a resource REQUEST to a WEB server to acquire a REQUEST sent by the browser;

normally, a REQUEST for acquiring a web page needs to be sent by inputting a URL into a browser, but this method simulates the operation of a user through a server, and a program must be used on the server to control the starting of the browser, the inputting of a web address and the closing of the browser. In this example, a browser is firstly installed at a server, the browser can select chrome, firefox and the like, and then a browser driver corresponding to the browser is installed at the server, the browser driver is provided by each browser manufacturer, the version of the browser driver is required to correspond to the version of the browser, otherwise, the browser cannot be driven to operate; the browser driver is operated through a program, and no interface is set when the browser starting parameters are configured, so that the browser can be operated conveniently by using a command line.

The server starts a browser and inputs the URL of the next-level webpage into the browser for access;

the browser monitor monitors a REQUEST sent by the browser when the browser accesses the next-level webpage URL, and when the browser is grabbed, the browser waits for three seconds to complete loading, otherwise, the grabbed REQUEST is not complete. Analyzing the domain name in the REQUEST, performing duplicate removal processing on the domain name, and storing the domain name in a domain name queue;

closing the browser listener and the browser;

resolving corresponding CNAME and A records on the DNS server for each domain name in the domain name queue, and setting CNAME to be empty if CNAME is not resolved. If the number of the A records corresponding to the domain name is more than two, summarizing all the A records into an A record list corresponding to the domain name.

Matching the A record corresponding to the domain name with an IP feature library, and judging the attribute of the A record; as shown in fig. 5, in this example, for a test URL whose flag bit is True, the IP feature library includes a BGP address table, a static address table, a cloud service address table, and an Education (EDU) address table, the matching order is the BGP address table, the static address table, the cloud service address table, and the Education (EDU) address table, when matching succeeds, the attribute of the record a is set as the name of the address table where the matching address is located, when matching between the address in the BGP address table and the static address table and the record a fails, and when matching succeeds with the cloud service address table, the attribute of the record a is set as a cloud service business name, and if matching fails in all the address tables, the attributes of the record a are set as interconnection. For the test URL with the flag bit of False, the IP feature library only comprises a cloud service address table and an Education (EDU) address, when the matching with the cloud service address table is successful, the attribute of the record A is set as a cloud service name, and if all the address tables are failed to match, the attribute of the record A is set as interconnection.

And writing the domain name in the domain name queue, the CNAME and A record corresponding to the domain name and the attribute of the A record into an EXCEL table.

And when all the next-level webpage URLs in the second-level URL queue count the data into the EXCEL form through the steps, returning the EXCEL form to the user side.

In the description herein, references to the description of the term "one embodiment," "some embodiments," "an example," "a specific example," or "some examples," etc., mean that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the invention. In this specification, the schematic representations of the terms do not necessarily refer to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

In light of the foregoing description of the preferred embodiment of the present invention, many modifications and variations will be apparent to those skilled in the art without departing from the spirit and scope of the invention. The technical scope of the present invention is not limited to the content of the specification, and must be determined according to the scope of the claims.

Claims (7)

1. The statistical analysis method for simulating the user behavior to dial and test the multistage domain names of the WEB sites is characterized by comprising the following steps of:

s1, establishing a local URL table, wherein the local URL table comprises at least one organization and a website name corresponding to the organization;

s2, extracting a test website name in the test URL, and referring to the local URL table by taking the test website name as a search object;

if the test website name is found in the local URL table, obtaining a test organization corresponding to the test website name from the local URL table, and generating an IP address feature library according to the test organization, wherein the IP address feature library comprises a BGP address table, a static address table and a cloud service address table;

if the test website name is not found in the local URL table, an IP address feature library is generated, wherein the IP address feature library comprises a cloud service address table;

s3, acquiring webpage source code content and a webpage source code encoding mode corresponding to the test URL, analyzing the webpage source code content by using the webpage source code encoding mode to obtain a next-stage webpage URL and storing the next-stage webpage URL into a second-stage URL queue; sequentially carrying out the steps S4-S10 on the next-level webpage URL in the second-level URL queue;

s4, starting a browser listener to monitor a REQUEST sent by a browser;

s5, starting a browser and inputting the URL of the next-level webpage into the browser for access;

s6, a browser monitor monitors a REQUEST sent when the browser accesses a next-level webpage URL, resolves a domain name in the REQUEST, and stores the domain name in a domain name queue after carrying out duplicate removal processing on the domain name;

s7, closing the browser monitor and the browser;

s8, resolving corresponding CNAME and A records for each domain name in the domain name queue;

s9, matching the record A corresponding to the domain name with an IP feature library, and judging the attribute of the record A;

and S10, writing the domain name in the domain name queue, the CNAME and A record corresponding to the domain name and the attribute of the A record into an EXCEL table.

2. The statistical analysis method for simulating the multilevel domain name of the user behavior dial-up WEB site according to claim 1, wherein:

in step S9, the record a corresponding to the domain name is sequentially matched with addresses in the BGP address table, the static address table, and the cloud service address table, and when matching is successful, the attribute of the record a is set to the name of the address table where the matching address is located, and if matching is failed, the attribute of the record a is set to interconnect.

3. The statistical analysis method for simulating the multilevel domain name of the user behavior dial-up WEB site according to claim 2, wherein: in said step S8, if the CNAME is not resolved, the CNAME is set to null.

4. The statistical analysis method for simulating the multilevel domain name of the user behavior dial-up WEB site according to claim 3, wherein: in step S8, if the number of a records corresponding to the domain name is two or more, all the a records are summarized into an a record list corresponding to the domain name.

5. The statistical analysis method for the multi-level domain name of the simulated user behavior dialing and testing WEB site according to claim 4, wherein: setting a flag bit corresponding to a test URL, in the step S2, if the test website name is found in the local URL table, setting the flag bit to True, and if the test website name is not found in the local URL table, setting the flag bit to False;

in the step S2, if the flag bit is True, a test organization corresponding to the test website name is obtained, and a BGP address table and a static address table corresponding to the test organization and used for generating an IP address feature library are obtained; and if the flag bit is False, using a cloud service address table corresponding to the test website name to generate the IP address feature library.

6. The statistical analysis method for the multi-level domain name of the simulated user behavior dialing and testing WEB site according to claim 5, wherein: further comprising the steps of:

before the step S2, the user enters a test interface after logging in, the test URL is obtained on the test interface and whether the test URL is reachable is tested, if yes, the step S2 is carried out, and if not, information indicating unreachability is returned to the user.

7. The statistical analysis method for the multi-level domain name of the simulated user behavior dialing and testing WEB site according to claim 6, wherein: further comprising the steps of:

and returning the EXCEL form to the user terminal.

Images