CN111898013A - Website monitoring method and device - Google Patents
Website monitoring method and device Download PDFInfo
- Publication number
- CN111898013A CN111898013A CN202010731482.3A CN202010731482A CN111898013A CN 111898013 A CN111898013 A CN 111898013A CN 202010731482 A CN202010731482 A CN 202010731482A CN 111898013 A CN111898013 A CN 111898013A
- Authority
- CN
- China
- Prior art keywords
- webpage
- monitored
- webpages
- monitoring
- layer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 263
- 238000000034 method Methods 0.000 title claims abstract description 38
- 230000009193 crawling Effects 0.000 claims abstract description 22
- 238000012163 sequencing technique Methods 0.000 claims abstract description 12
- 230000006870 function Effects 0.000 claims description 44
- 238000012545 processing Methods 0.000 claims description 21
- 238000010586 diagram Methods 0.000 description 16
- 238000004590 computer program Methods 0.000 description 7
- 230000008030 elimination Effects 0.000 description 4
- 238000003379 elimination reaction Methods 0.000 description 4
- 230000000694 effects Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- ZXQYGBMAQZUVMI-GCMPRSNUSA-N gamma-cyhalothrin Chemical compound CC1(C)[C@@H](\C=C(/Cl)C(F)(F)F)[C@H]1C(=O)O[C@H](C#N)C1=CC=CC(OC=2C=CC=CC=2)=C1 ZXQYGBMAQZUVMI-GCMPRSNUSA-N 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a method and a device for monitoring a website, which comprise the following steps: acquiring web pages of a website to be monitored, determining key indexes of the web pages according to the crawling depth of a web crawler and/or the web page updating time recorded by a search engine, determining the key degrees of the web pages according to the key indexes of the web pages, and sequencing; and then, placing the sorted webpages into a preset number of layers of to-be-monitored webpage pyramids, determining monitoring periods of all layers in the to-be-monitored webpage pyramids and monitoring webpage quantities corresponding to the monitoring periods of all layers according to a preset reference monitoring period, and monitoring the webpages of all layers in the to-be-monitored webpage pyramids in a sliding monitoring webpage mode, so that monitoring of different monitoring periods is carried out on the webpages with different criticalities, the periods and the quantities of the monitored webpages are adjusted in a self-adaptive mode, and the resource consumption and the cost of the monitored webpages are reduced.
Description
Technical Field
The invention relates to the field of network security, in particular to a method and a device for monitoring a website.
Background
The website monitoring provides services such as website loophole, website trojan real-time monitoring, website tampering real-time monitoring and the like. For a given website, capturing webpages of the website by a method such as a web crawler and the like, then determining a target webpage to be monitored according to the captured webpages, and monitoring the target webpage according to a preset frequency.
In the prior art, the real-time performance requirements of webpage Trojan horse hanging monitoring and webpage information tampering monitoring are high, and taking webpage information tampering monitoring as an example, a user wants to monitor a large number of webpages with high frequency, and finds abnormal tampering data of a certain webpage at the first time. The monitoring service provider may wish to reduce the number and frequency of web pages monitored, thereby reducing resource consumption and cost. The number and the frequency of the monitoring webpages are determined by a parameter configuration mode in the conventional method, but the method has certain randomness and cannot self-adaptively adjust the number and the frequency of the monitoring webpages.
Disclosure of Invention
The embodiment of the invention provides a website monitoring method and device, which are used for realizing self-adaptive adjustment of the number and frequency of monitoring webpages and improving the monitoring range.
In a first aspect, an embodiment of the present invention provides a method for monitoring a website, including:
acquiring a webpage of a website to be monitored; the webpage is obtained by crawling by a web crawler and/or recording by a search engine;
determining key indexes of all the webpages according to the crawling depth of the web crawler and/or the webpage updating time recorded by a search engine;
determining the criticality of each webpage according to the key index of each webpage, and sequencing; putting the sorted webpages into a webpage pyramid to be monitored with a preset number of layers; the number of the webpages put into each layer in the webpage pyramid to be monitored is determined by a preset index function;
determining the monitoring period of each layer in the pyramid of the web pages to be monitored and the number of the monitoring web pages corresponding to the monitoring period of each layer according to a preset reference monitoring period;
monitoring the webpages in each layer in the webpage pyramid to be monitored in a sliding mode according to the monitoring periods of each layer in the webpage pyramid to be monitored and the number of the monitored webpages corresponding to the monitoring periods of each layer.
According to the technical scheme, the webpages are sorted according to the criticality of the webpages, the webpages are sequentially stored into the webpage pyramid to be monitored with the preset number of layers according to the sorting result, the monitoring period of each layer is determined according to the preset reference monitoring period, the period of monitoring webpages of each layer in the webpage pyramid to be monitored is adjusted in a self-adaptive mode, the number of monitoring webpages of each layer in the webpage pyramid to be monitored is adjusted in a self-adaptive mode according to the number of the monitoring webpages of each layer of the determined webpage pyramid to be monitored, and the resource consumption and the cost of the monitoring webpages are reduced. And different quantities of webpage monitoring can be carried out on each layer of webpage in the pyramid of the webpage to be monitored in a self-adaptive manner under the condition of limited resource consumption.
Optionally, the determining the key index of each web page according to the crawling depth of the web crawler and/or the update time of the web pages included in the search engine includes:
determining the relation between a father page and a son page in each webpage according to the crawling depth of the breadth-first method of the web crawler; determining a first key index of each webpage according to the relationship between a father page and a son page in each webpage; the sum of the first key index of the parent page and a first preset value is the first key index of the child page; and/or
Obtaining the time difference between the webpage updating time of each webpage and the current time according to the webpage updating time recorded in the search engine of each webpage, and determining a second key index of each webpage according to the time difference;
the first key index of each webpage and the second key index of each webpage are subjected to duplication elimination to obtain the key index of each webpage;
wherein the time difference and the second key index conform to equation (1);
the formula (1) is:
wherein KI(Δt)The second key index of each webpage is obtained; and delta t is the time difference between the webpage updating time of each webpage and the current time.
In the technical scheme, two methods are used for determining the key indexes of the webpages, on one hand, the parent-child page relation among the webpages is determined according to the breadth priority of the web crawler, and the first key index of each webpage is determined according to the parent-child page relation among the webpages; on the other hand, the second key index of each webpage is determined according to the time difference between the webpage updating time of each webpage and the current time, and then the key index of each webpage is obtained after duplication removal, so that the accuracy of determining the key index of each webpage is improved, and the webpage monitoring effect is further improved.
Optionally, the removing the duplicate of the first key index of each web page and the second key index of each web page to obtain the key index of each web page includes:
if the first key index and the second key index exist in the webpage, judging whether the first key index is larger than the second key index, if so, determining the first key index as the key index of the webpage, otherwise, determining the second key index as the key index of the webpage;
and if the first key index or the second key index exists in the webpage, determining the first key index or the second key index as the key index of the webpage.
According to the technical scheme, whether the first key index and the second key index obtained through two modes exist in the webpage or not is judged, if yes, duplication is removed through judging the size of the first key index and the second key index, the final key index of the webpage is further determined, the accuracy of determining the key indexes of the webpages is improved, and the webpage monitoring effect is further improved.
Optionally, the determining the criticality of each web page according to the key index of each web page and ranking the criticality of each web page include:
determining the reciprocal of the key index of each webpage as the key degree of each webpage;
and sequencing the webpages according to the sequence of the criticality from large to small according to the criticality of the webpages.
According to the technical scheme, the criticality of each webpage is determined through the key index, the webpages are sequenced according to the criticality, the corresponding monitoring period is set according to the criticality of the webpages, and the period of monitoring the webpages is adjusted in a self-adaptive mode.
Optionally, the determining, by a preset index function, the number of the webpages put into each layer in the pyramid of the webpages to be monitored includes:
determining a scale coefficient of the website according to the total number of the webpages of the website to be monitored and a second preset value number of the webpages which can be stored in the webpage pyramid to be monitored;
determining the number of the webpages put into each layer in the pyramid of the webpages to be monitored according to the scale coefficient and a preset index function;
the scale factor conforms to equation (2);
the formula (2) is:
wherein alpha is the scale coefficient of the website to be monitored; stotalThe number of web pages that are the web site; s is the number of second preset values of the web pages which can be stored in the web page pyramid to be monitored;
the number of the webpages put into each layer in the webpage pyramid to be monitored accords with a formula (3);
the formula (3) is:
Sl＝α*2l……………………………………………………………(3)；
wherein S islThe number of web pages which can be stored in the l layer of the pyramid of the web pages to be monitored; alpha is the scale coefficient of the website; 2lIs a preset exponential function; and l is the l-th layer of the webpage pyramid to be monitored.
In the technical scheme, the number of the webpages put into each layer in the pyramid of the webpages to be monitored is determined according to the scale coefficient of the websites to be monitored and the second preset value number of the webpages which can be stored in the pyramid of the webpages to be monitored, so that the webpages of the websites to be monitored can be completely stored in the pyramid of the webpages to be monitored, and the monitoring effect of the webpages of the websites to be monitored is improved.
Optionally, the determining the monitoring periods of the layers in the pyramid of the web pages to be monitored and the number of the monitoring web pages corresponding to the monitoring periods of the layers includes:
determining the monitoring period of each layer in the pyramid of the webpage to be monitored according to the preset reference monitoring period and the preset index function;
determining the number of monitoring webpages corresponding to the monitoring period of each layer according to the number of layers of the webpage pyramid to be monitored and the preset index function;
the monitoring period of each layer in the pyramid of the webpage to be monitored accords with a formula (4);
the formula (4) is:
Tl＝Tbasic*2l………………………………………………………(4)；
wherein, TlThe monitoring period of the l layer in the pyramid of the webpage to be monitored is set; t isbasicThe preset reference monitoring period is set; 2lIs a preset exponential function; l is the l-th layer of the webpage pyramid to be monitored;
the number of monitoring web pages corresponding to the monitoring period of each layer accords with a formula (5);
the formula (5) is:
Ptl＝2l…………………………………………………………(5)；
wherein, PtlThe number of monitoring web pages corresponding to the monitoring period of each layer; 2lIs a preset exponential function; and l is the l-th layer of the webpage pyramid to be monitored.
According to the technical scheme, the monitoring period of each layer is determined according to the preset reference monitoring period, monitoring of different monitoring periods is carried out on the webpages with different layers in the webpage pyramid to be monitored, the period of the monitoring webpages is adjusted in a self-adaptive mode, the number of the monitoring webpages with different layers in the webpage pyramid to be monitored is adjusted in a self-adaptive mode according to the number of the monitoring webpages with each layer, and webpage monitoring of different numbers is carried out on the webpages with each layer in the webpage pyramid to be monitored in a self-adaptive mode under the condition of limited resource consumption. The resource consumption and the cost of monitoring the webpage are reduced.
Optionally, the monitoring the webpages in each layer in the webpage pyramid to be monitored in a sliding manner according to the monitoring periods of each layer in the webpage pyramid to be monitored and the number of the monitored webpages corresponding to the monitoring periods of each layer includes:
determining the number of the sliding webpages per monitoring period of each layer in the webpage pyramid to be monitored;
selecting webpages from each layer in the pyramid of the webpages to be monitored for monitoring according to the number of the webpages sliding in each monitoring period of each layer and the number of the monitoring webpages corresponding to the monitoring period of each layer;
the number of the sliding webpages of each layer per monitoring period accords with a formula (6);
the formula (6) is:
W＝2l-1……………………………………………………………(6)；
wherein, W is the number of the sliding webpages in each monitoring period of the first layer of the webpage pyramid to be monitored; and l is the l-th layer of the webpage pyramid to be monitored.
According to the technical scheme, the number of the sliding webpages per monitoring period determined by each layer in the pyramid of the webpages to be monitored is determined, so that each corresponding layer can be monitored from different webpages each time, the webpages of each layer can be monitored alternately in a self-adaptive manner, and the webpage monitoring range is enlarged.
In a second aspect, an embodiment of the present invention provides a website monitoring apparatus, including:
the acquisition module is used for acquiring a webpage of a website to be monitored; the webpage is obtained by crawling by a web crawler and/or recording by a search engine;
the processing module is used for determining the key index of each webpage according to the crawling depth of the web crawler and/or the webpage updating time recorded by the search engine;
determining the criticality of each webpage according to the key index of each webpage, and sequencing; putting the sorted webpages into a webpage pyramid to be monitored with a preset number of layers; the number of the webpages put into each layer in the webpage pyramid to be monitored is determined by a preset index function;
determining the monitoring period of each layer in the pyramid of the web pages to be monitored and the number of the monitoring web pages corresponding to the monitoring period of each layer according to a preset reference monitoring period;
monitoring the webpages in each layer in the webpage pyramid to be monitored in a sliding mode according to the monitoring periods of each layer in the webpage pyramid to be monitored and the number of the monitored webpages corresponding to the monitoring periods of each layer.
Optionally, the processing module is specifically configured to:
determining the relation between a father page and a son page in each webpage according to the crawling depth of the breadth-first method of the web crawler; determining a first key index of each webpage according to the relationship between a father page and a son page in each webpage; the sum of the first key index of the parent page and a first preset value is the first key index of the child page; and/or
Obtaining the time difference between the webpage updating time of each webpage and the current time according to the webpage updating time recorded in the search engine of each webpage, and determining a second key index of each webpage according to the time difference;
the first key index of each webpage and the second key index of each webpage are subjected to duplication elimination to obtain the key index of each webpage;
wherein the time difference and the second key index conform to equation (1);
the formula (1) is:
wherein KI(Δt)The second key index of each webpage is obtained; and delta t is the time difference between the webpage updating time of each webpage and the current time.
Optionally, the processing module is specifically configured to:
if the first key index and the second key index exist in the webpage, judging whether the first key index is larger than the second key index, if so, determining the first key index as the key index of the webpage, otherwise, determining the second key index as the key index of the webpage;
and if the first key index or the second key index exists in the webpage, determining the first key index or the second key index as the key index of the webpage.
Optionally, the processing module is specifically configured to:
determining the reciprocal of the key index of each webpage as the key degree of each webpage;
and sequencing the webpages according to the sequence of the criticality from large to small according to the criticality of the webpages.
Optionally, the processing module is specifically configured to:
determining a scale coefficient of the website according to the total number of the webpages of the website to be monitored and a second preset value number of the webpages which can be stored in the webpage pyramid to be monitored;
determining the number of the webpages put into each layer in the pyramid of the webpages to be monitored according to the scale coefficient and a preset index function;
the scale factor conforms to equation (2);
the formula (2) is:
wherein alpha is the scale coefficient of the website; stotalThe number of web pages that are the web site; s is the number of second preset values of the web pages which can be stored in the web page pyramid to be monitored;
the number of the webpages put into each layer in the webpage pyramid to be monitored accords with a formula (3);
the formula (3) is:
Sl＝α*2l……………………………………………………………(3)；
wherein S islThe number of web pages which can be stored in the l layer of the pyramid of the web pages to be monitored; alpha is the scale coefficient of the website; 2lIs a preset exponential function; and l is the l-th layer of the webpage pyramid to be monitored.
Optionally, the processing module is specifically configured to:
determining the monitoring period of each layer in the pyramid of the webpage to be monitored according to the preset reference monitoring period and the preset index function;
determining the number of monitoring webpages corresponding to the monitoring period of each layer according to the number of layers of the webpage pyramid to be monitored and the preset index function;
the monitoring period of each layer in the pyramid of the webpage to be monitored accords with a formula (4);
the formula (4) is:
Tl＝Tbasic*2l………………………………………………………(4)；
wherein, TlThe monitoring period of the l layer in the pyramid of the webpage to be monitored is set; t isbasicThe preset reference monitoring period is set; 2lIs a preset exponential function; l is the l-th layer of the webpage pyramid to be monitored;
the number of monitoring web pages corresponding to the monitoring period of each layer accords with a formula (5);
the formula (5) is:
Ptl＝2l…………………………………………………………(5)；
wherein, PtlThe number of monitoring web pages corresponding to the monitoring period of each layer; 2lIs a preset exponential function; and l is the l-th layer of the webpage pyramid to be monitored.
Optionally, the processing module is specifically configured to:
determining the number of the sliding webpages per monitoring period of each layer in the webpage pyramid to be monitored;
selecting webpages from each layer in the pyramid of the webpages to be monitored for monitoring according to the number of the webpages sliding in each monitoring period of each layer and the number of the monitoring webpages corresponding to the monitoring period of each layer;
the number of the sliding webpages of each layer per monitoring period accords with a formula (6);
the formula (6) is:
W＝2l-1……………………………………………………………(6)；
wherein, W is the number of the sliding webpages in each monitoring period of the first layer of the webpage pyramid to be monitored; and l is the l-th layer of the webpage pyramid to be monitored.
In a third aspect, an embodiment of the present invention further provides a computing device, including:
a memory for storing program instructions;
and the processor is used for calling the program instruction stored in the memory and executing the website monitoring method according to the obtained program.
In a fourth aspect, the embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are configured to enable a computer to execute the above website monitoring method.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings needed to be used in the description of the embodiments will be briefly introduced below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without creative efforts.
FIG. 1 is a system architecture diagram according to an embodiment of the present invention;
fig. 2 is a flowchart illustrating a website monitoring method according to an embodiment of the present invention;
FIG. 3 is a diagram illustrating a functional image according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of a pyramid of a webpage to be monitored according to an embodiment of the present invention;
fig. 5 is a schematic diagram of web page monitoring according to an embodiment of the present invention;
fig. 6 is a schematic structural diagram of a website monitoring apparatus according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer, the present invention will be described in further detail with reference to the accompanying drawings, and it is apparent that the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Fig. 1 illustrates an exemplary system architecture to which embodiments of the present invention are applicable, which includes a server 100, and the server 100 may include a processor 110 and a memory 120.
The processor 110 is a control center of the server 100, connects each part of the whole server 100 by using various interfaces and routes, obtains the web pages of the website to be monitored according to the crawling depth of the web crawler and/or the recording of a search engine by executing software programs and/or modules stored in the memory 120, calls data stored in the memory 120, executes various functions and processes data of the server 100, and monitors the web pages of each layer in the pyramid of the web pages to be monitored according to the monitoring period of each layer in the pyramid of the web pages to be monitored and the number of the monitoring web pages corresponding to the monitoring period of each layer. Alternatively, processor 110 may include one or more processing units.
The memory 120 may be used to store software programs, modules and web pages of the website to be monitored, and the processor 110 executes various functional applications and data processing by running the software programs and modules stored in the memory 120. The memory 120 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program required for at least one function, and the like; the storage data area may store data created according to a business process, and the like. Further, the memory 120 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device.
It should be noted that the structure shown in fig. 1 is only an example, and the embodiment of the present invention is not limited thereto.
Based on the above description, fig. 2 exemplarily shows a flow of a website monitoring method according to an embodiment of the present invention, where the flow may be performed by a website monitoring apparatus.
As shown in fig. 2, the process specifically includes:
In the embodiment of the invention, the webpage is obtained by crawling by a web crawler and/or recording by a search engine. For example, a web crawler is used to crawl a web page in a website to be monitored, or a web page corresponding to a URL (uniform resource Locator) is obtained according to the URL received and recorded by a search engine.
The URL may be understood as a link corresponding to a web page.
In the embodiment of the invention, the key index of the webpage can be determined in two ways, including a web crawler crawling way and the webpage update time recorded by a search engine.
Further, determining the relationship between a parent page and a child page in each webpage according to the crawling depth of the breadth-first method of the web crawler; determining a first key index of each webpage according to the relationship between a father page and a son page in each webpage; the sum of the first key index of the parent page and a first preset value is the first key index of the child page; and/or
Obtaining the time difference between the webpage updating time of each webpage and the current time according to the webpage updating time recorded in the search engine of each webpage, and determining a second key index of each webpage according to the time difference;
the first key index of each webpage and the second key index of each webpage are subjected to duplication elimination to obtain the key index of each webpage;
wherein the time difference and the second key index conform to formula (1);
the formula (1) is:
wherein KI(Δt)The second key index of each webpage; and delta t is the time difference between the webpage updating time of each webpage and the current time.
In the embodiment of the invention, on one hand, the breadth-first method is also called breadth-first, which means that a web crawler can capture all the web pages of an initial web page, then capture the corresponding web pages in each initial web page according to the links in each initial web page, and capture the web pages in sequence to obtain the relationship between the parent pages and the child pages of each web page, namely the depth relationship between the web pages. And then determining a first key index of each webpage according to the obtained relationship between the parent page and the child page of each webpage and a first preset value. For example, a first key index of a preset parent page (i.e., a starting web page) is 1 (a first preset value), then the first key index of a child page corresponding to the parent page is determined to be the sum of the first key index of the parent page and the first preset value, that is, the first key index of the child page corresponding to the preset parent page is 2, and in turn, the first key index of the child page corresponding to the child page is 3, so as to obtain the first key index of each web page.
It should be noted that the first preset value may be a value preset empirically or a value determined according to the depth of the web page. For example, the depth of the web pages is reflected by the relationship of the web page levels, the depth of the first web page is 1, the depth of the second layer of web pages is 2, and the depth of the third layer of web pages is 3, so that the depth of each layer of web pages is determined by adding 1 to the depth of the first web page, namely 1 is a first preset value.
On the other hand, the web pages included in the search engine are time difference-labeled, where the time difference is a difference between the web page update time of each web page and the current time, and for example, if the web page update time of the a web page is 0/7/15/2020 and the current time is 7/16/2020, the time difference of the a web page is 7 hours ago. And then determining a second key index of each webpage according to the time difference and the formula 3. In order to better explain the above technical solutions, the following will be illustrated in specific examples.
Example 1
Inputting keywords into a search engine to obtain corresponding content of a page, extracting links in the page to obtain a time difference corresponding to the page, then obtaining a second key index corresponding to the time difference according to the formula (1) to further obtain a second key index corresponding to the page, and as shown in fig. 3, an exemplary schematic diagram of a function image is shown, where as shown in fig. 3, when an abscissa is 0 (that is, the page is updated at the current moment), the second key index corresponding to the page is 1, and when the abscissa is 24, the second key index corresponding to the page is 3. It should be noted that, in fig. 3, the abscissa indicates the time difference of the web page, and the ordinate indicates the second key index corresponding to the web page.
Illustratively, the first key index and/or the second key index of the web pages are obtained through the technical scheme, and then the first key index and the second key index of each web page are deduplicated to obtain the final used key index of each web page.
Specifically, if the first key index and the second key index exist in the webpage, whether the first key index is larger than the second key index is judged, if yes, the first key index is determined as the key index of the webpage, and if not, the second key index is determined as the key index of the webpage;
and if the first key index or the second key index exists in the webpage, determining the first key index or the second key index as the key index of the webpage.
According to the embodiment of the invention, the first key index and the second key index obtained by the time difference method may exist in the web page in the website to be monitored, namely the first key index and the second key index exist in the same web page, and the key index of the web page is determined by selecting the index value to be large.
For example, if the first key index and the second key index of the web page are equal, the second key index is selected to be determined as the key index of the web page.
For the technical scheme of the invention, for different use scenes, the key index of the webpage can be determined by selecting the small index value. If the first key index and the second key index of the webpage are equal, the first key index can be selected to be determined as the key index of the webpage.
And if only the first key index or the second key index exists in the webpage, determining one of the first key index and the second key index as the key index of the webpage.
In the embodiment of the invention, the key corresponding to the webpage is determined according to the determined key index corresponding to the webpage, so that the webpage is sorted according to the key degree.
Further, determining the reciprocal of the key index of each webpage as the key degree of each webpage;
and sequencing the webpages according to the sequence of the criticality from large to small according to the criticality of the webpages.
In the embodiment of the present invention, it is determined that the key index and the key degree of each web page are reciprocal, so that the key degree of each web page is obtained through the key index of each web page, and the web pages are sorted according to the descending order, for example, if the key index KI of a web page is 2, the key index KI of b web page is 4, and the key index KI of c web page is 6, the key degree K of a web page is 1/2, the key degree K of b web page is 1/4, and the key degree K of c web page is 1/6. Then the ordered web page sequence is: a. b and c.
Illustratively, after the webpages are sorted, a preset number of layers of the pyramid of the webpages to be monitored are put in.
It should be noted that the number of the web pages placed in each layer of the pyramid of the web pages to be monitored is determined by a preset index function.
Further, determining a scale coefficient of the website according to the total number of the webpages of the website to be monitored and a second preset value number of the storable webpages of the webpage pyramid of the website to be monitored;
determining the number of the webpages put into each layer in the pyramid of the webpages to be monitored according to the scale coefficient and a preset index function;
the scale factor conforms to equation (2);
the formula (2) is:
wherein, alpha is the scale coefficient of the website; stotalThe number of web pages that are web sites; s is the number of second preset values of the web pages which can be stored in the web page pyramid to be monitored;
the number of the webpages put into each layer in the webpage pyramid to be monitored accords with a formula (3);
the formula (3) is:
Sl＝α*2l……………………………………………………………(3)；
wherein S islThe number of web pages which can be stored in the first layer of the web page pyramid to be monitored; alpha is the scale coefficient of the website; 2lIs a preset exponential function; l is the first layer of the pyramid of the web page to be monitored.
According to the embodiment of the invention, whether the total number of the pages of the currently monitored website is greater than a preset second preset value or not is judged, if yes, a scale coefficient is determined, and the number of the pages put into each layer in the pyramid of the web pages to be monitored is determined, so that all the web pages in the website to be monitored are stored in the pyramid of the web pages to be monitored. In order to better explain the above technical solutions, the following will be illustrated in specific examples.
Example 2
And if the number S of the second preset values of the web pages storable in the web page pyramid to be monitored is 1023 pages, and the total number of the web pages of the currently monitored website is 1534, determining that the scale coefficient alpha is 1.5 according to the formula (2). And then determining the number of the webpages put into each layer in the pyramid of the webpages to be monitored according to a formula (3). Fig. 4 exemplarily shows a schematic diagram of a pyramid of web pages to be monitored, as shown in fig. 4, the number of web pages placed in the 0 th layer of the pyramid of web pages to be monitored is 1, the number of web pages placed in the 2 nd layer is 3, and so on, by combining with formula 3, it can be known that the number of web pages placed in the l (not 0) th layer is (S)l-1)-(Sl-1)。
And 204, determining the monitoring period of each layer in the pyramid of the web pages to be monitored and the number of the monitoring web pages corresponding to the monitoring period of each layer according to a preset reference monitoring period.
In the embodiment of the invention, the monitoring period of each layer in the pyramid of the webpage to be monitored is determined according to the preset reference monitoring period, and the number of the monitoring webpages corresponding to the monitoring period of each layer is determined according to the number of the layers of the pyramid of the webpage to be monitored.
Specifically, the monitoring period of each layer in the pyramid of the webpage to be monitored is determined according to a preset reference monitoring period and a preset index function;
determining the number of monitoring webpages corresponding to the monitoring period of each layer according to the number of layers of the webpage pyramid to be monitored and a preset index function;
the monitoring period of each layer in the pyramid of the webpage to be monitored accords with a formula (4);
equation (4) is:
Tl＝Tbasic*2l………………………………………………………(4)；
wherein, TlThe monitoring period of the first layer in the pyramid of the webpage to be monitored is set; t isbasicA preset reference monitoring period; 2lIs a preset exponential function; l is the first layer of the webpage pyramid to be monitored;
the number of monitoring web pages corresponding to the monitoring period of each layer accords with a formula (5);
equation (5) is:
Ptl＝2l…………………………………………………………(5)；
wherein, PtlMonitoring the number of web pages corresponding to the monitoring period of each layer; 2lIs a preset exponential function; l is the first layer of the pyramid of the web page to be monitored.
According to the embodiment of the invention, the monitoring period of each layer in the pyramid of the webpage to be monitored is calculated through the preset monitoring period and the preset index function so as to distinguish the monitoring periods of the webpages in the pyramid of the webpage to be monitored. For example, if the preset monitoring period is 2/second, that is, monitoring is performed once in 2 seconds, according to formula (4), it is determined that the monitoring period of the 0 th layer in the pyramid of the web page to be monitored is 2/second, the monitoring period of the 1 st layer is 4/second, and the monitoring period of the … … 9 th layer is 1024/second.
It should be noted that the unit of the preset detection period may be x/month, x/day, x/hour, x/minute, or x/second, where x is a number other than 0, and is not limited herein.
Illustratively, the number of the monitored webpages corresponding to each layer of the pyramid of the webpages to be monitored is determined according to the number of layers of the pyramid of the webpages to be monitored and a preset index function, so that a certain number of webpages are adaptively selected for monitoring during monitoring, not all webpages are monitored, and resource loss is reduced. For example, according to the formula (5), the number of monitoring webpages in the layer 0 is 1, the number of monitoring webpages in the layer 1 is 2, the number of monitoring webpages in the layer 2 is 4, … …, and the number of monitoring webpages in the layer 9 is 512, that is, the webpages are monitored from the starting position of each layer, the monitoring number is the number of monitoring webpages corresponding to the monitoring period, and is also the number of monitoring webpages corresponding to each layer, as shown in fig. 4, and the shaded portion in fig. 4 is the monitoring webpages, such as 1, 2 … … 1278.
In the embodiment of the invention, the number of the webpages sliding in each monitoring period of each layer in the webpage pyramid to be monitored is determined, and the webpage position of each layer for monitoring the webpages is determined according to the number of the webpages sliding in each monitoring period of each layer, so that the webpages in each layer in the webpage pyramid to be monitored are alternately monitored.
Further, determining the number of the sliding webpages per monitoring period of each layer in the webpage pyramid to be monitored;
selecting webpages from each layer in the pyramid of the webpages to be monitored for monitoring according to the number of the webpages sliding in each monitoring period of each layer and the number of the monitoring webpages corresponding to the monitoring period of each layer;
the number of the sliding web pages of each layer in each monitoring period accords with a formula (6);
equation (6) is:
W＝2l-1……………………………………………………………(6)；
wherein, W is the number of the sliding webpages in each monitoring period of the first layer of the webpage pyramid to be monitored; l is the first layer of the pyramid of the web page to be monitored.
According to the embodiment of the invention, aiming at the web pages in the same layer, the web pages are slid according to each corresponding monitoring period in the layerThe number of the moving web pages is changed after each monitoring period, so that the web pages in the layer are monitored in a rotation mode. For example, fig. 5 exemplarily shows a schematic diagram of web page monitoring, and as shown in fig. 5, the number of web pages sliding per monitoring period of the layer 3 in the pyramid of web pages to be monitored is determined to be W ═ 23-1＝4。
In the first monitoring period, the number of the monitored webpages in the 3 rd layer in the pyramid of the webpages to be monitored is 11-18, in the second monitoring period, 4 webpages slide backwards for monitoring, and the number of the monitored webpages is 15-22, and similarly, in the third monitoring period, the number of the monitored webpages is 19-22 and 11-14. Note that, the shaded portion in fig. 5 is a monitored web page.
In the embodiment of the invention, the monitoring cost of the website to be monitored can be calculated through the technical scheme.
Illustratively, the scale of the website to be monitored is determined according to the total number of the webpages of the website to be monitored, and the cost of the website to be monitored is determined according to unit prices of the preset websites with different scales in the minimum monitoring period and the preset reference monitoring period in the technical scheme.
The scale of the website to be monitored is divided into a small website, a medium website and a large website through a first scale preset value and a second scale preset value. Specifically, if the website to be monitored is not larger than a first scale preset value, the website to be monitored is a small website, if the website to be monitored is larger than the first scale preset value and not larger than a second scale preset value, the website to be monitored is a medium website, and if the website to be monitored is larger than the second scale preset value, the website to be monitored is a large website;
the cost of the website to be monitored conforms to the formula (7);
equation (7) is:
wherein C is the cost of the website to be monitored; csIs a small netUnit price at minimum monitoring period of a station; cmUnit price under the minimum monitoring period of the medium-sized website; clThe unit price of the large website in the minimum monitoring period; t isminA preset minimum monitoring period of the website; t isbasicThe monitoring period is a preset reference.
In order to better explain the above technical solutions, the following examples are given.
Example 3
Preset minimum monitoring period T min1/s, a preset reference monitoring period T basic2/s, unit price C at minimum monitoring period for small web sitess1000, unit price C at minimum monitoring period for medium-sized websitem2000, unit price C at minimum monitoring period for large web sitesl3000, the corresponding website monitoring cost is 500/1000/1500.
It should be noted that the time of the unit price at the minimum monitoring period may be one month, one year, etc., and is not limited in this embodiment.
According to the embodiment of the invention, the monitoring web pages and the monitoring period in the web pages are determined by presetting the pyramid of the web pages to be monitored, the monitoring cost is linearized, and the criticality of each web page is quantized according to the key indexes, so that the monitoring period and the monitoring number are adaptively determined, for example, 1023 web pages exist in a certain website, and the preset reference monitoring period is 5min, so that according to the technical scheme of the invention, the monitoring times of the web pages in the pyramid of the web pages to be monitored are 2880 within 24 hours, and the minimum monitoring period in the web pages is 5min, namely the web pages are monitored for 288 times, and similarly, the web pages with the maximum monitoring period are only monitored for 1 time. And finally, adjusting and rotating the monitoring web pages of the corresponding layers through the periods corresponding to the layers in the pyramid of the web pages to be monitored, so as to solve the problem of incomplete coverage of the monitoring target.
Based on the same technical concept, fig. 6 exemplarily shows a structure of an apparatus for website monitoring, which may perform a flow of a method for website monitoring according to an embodiment of the present invention.
As shown in fig. 6, the apparatus specifically includes:
an obtaining module 601, configured to obtain a webpage of a website to be monitored; the webpage is obtained by crawling by a web crawler and/or recording by a search engine;
the processing module 602 is configured to determine a key index of each web page according to a crawling depth of the web crawler and/or a web page update time included in a search engine;
determining the criticality of each webpage according to the key index of each webpage, and sequencing; putting the sorted webpages into a webpage pyramid to be monitored with a preset number of layers; the number of the webpages put into each layer in the webpage pyramid to be monitored is determined by a preset index function;
determining the monitoring period of each layer in the pyramid of the web pages to be monitored and the number of the monitoring web pages corresponding to the monitoring period of each layer according to a preset reference monitoring period;
monitoring the webpages in each layer in the webpage pyramid to be monitored in a sliding mode according to the monitoring periods of each layer in the webpage pyramid to be monitored and the number of the monitored webpages corresponding to the monitoring periods of each layer.
Optionally, the processing module 602 is specifically configured to:
determining the relation between a father page and a son page in each webpage according to the crawling depth of the breadth-first method of the web crawler; determining a first key index of each webpage according to the relationship between a father page and a son page in each webpage; the sum of the first key index of the parent page and a first preset value is the first key index of the child page; and/or
Obtaining the time difference between the webpage updating time of each webpage and the current time according to the webpage updating time recorded in the search engine of each webpage, and determining a second key index of each webpage according to the time difference;
the first key index of each webpage and the second key index of each webpage are subjected to duplication elimination to obtain the key index of each webpage;
wherein the time difference and the second key index conform to equation (1);
the formula (1) is:
wherein KI(Δt)The second key index of each webpage is obtained; and delta t is the time difference between the webpage updating time of each webpage and the current time.
Optionally, the processing module 602 is specifically configured to:
if the first key index and the second key index exist in the webpage, judging whether the first key index is larger than the second key index, if so, determining the first key index as the key index of the webpage, otherwise, determining the second key index as the key index of the webpage;
and if the first key index or the second key index exists in the webpage, determining the first key index or the second key index as the key index of the webpage.
Optionally, the processing module 602 is specifically configured to:
determining the reciprocal of the key index of each webpage as the key degree of each webpage;
and sequencing the webpages according to the sequence of the criticality from large to small according to the criticality of the webpages.
Optionally, the processing module 602 is specifically configured to:
determining a scale coefficient of the website according to the total number of the webpages of the website to be monitored and a second preset value number of the webpages which can be stored in the webpage pyramid to be monitored;
determining the number of the webpages put into each layer in the pyramid of the webpages to be monitored according to the scale coefficient and a preset index function;
the scale factor conforms to equation (2);
the formula (2) is:
wherein alpha is the scale coefficient of the website; stotalThe number of web pages that are the web site; s is the number of second preset values of the web pages which can be stored in the web page pyramid to be monitored;
the number of the webpages put into each layer in the webpage pyramid to be monitored accords with a formula (3);
the formula (3) is:
Sl＝α*2l……………………………………………………………(3)；
wherein S islThe number of web pages which can be stored in the l layer of the pyramid of the web pages to be monitored; alpha is the scale coefficient of the website; 2lIs a preset exponential function; and l is the l-th layer of the webpage pyramid to be monitored.
Optionally, the processing module 602 is specifically configured to:
determining the monitoring period of each layer in the pyramid of the webpage to be monitored according to the preset reference monitoring period and the preset index function;
determining the number of monitoring webpages corresponding to the monitoring period of each layer according to the number of layers of the webpage pyramid to be monitored and the preset index function;
the monitoring period of each layer in the pyramid of the webpage to be monitored accords with a formula (4);
the formula (4) is:
Tl＝Tbasic*2l………………………………………………………(4)；
wherein, TlThe monitoring period of the l layer in the pyramid of the webpage to be monitored is set; t isbasicThe preset reference monitoring period is set; 2lIs a preset exponential function; l is the l-th layer of the webpage pyramid to be monitored;
the number of monitoring web pages corresponding to the monitoring period of each layer accords with a formula (5);
the formula (5) is:
Ptl＝2l…………………………………………………………(5)；
wherein, PtlThe number of monitoring web pages corresponding to the monitoring period of each layer; 2lIs a preset exponential function; and l is the l-th layer of the webpage pyramid to be monitored.
Optionally, the processing module 602 is specifically configured to:
determining the number of the sliding webpages per monitoring period of each layer in the webpage pyramid to be monitored;
selecting webpages from each layer in the pyramid of the webpages to be monitored for monitoring according to the number of the webpages sliding in each monitoring period of each layer and the number of the monitoring webpages corresponding to the monitoring period of each layer;
the number of the sliding webpages of each layer per monitoring period accords with a formula (6);
the formula (6) is:
W＝2l-1……………………………………………………………(6)；
wherein, W is the number of the sliding webpages in each monitoring period of the first layer of the webpage pyramid to be monitored; and l is the l-th layer of the webpage pyramid to be monitored.
Based on the same technical concept, an embodiment of the present invention further provides a computing device, including:
a memory for storing program instructions;
and the processor is used for calling the program instruction stored in the memory and executing the website monitoring method according to the obtained program.
Based on the same technical concept, the embodiment of the invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used for enabling a computer to execute the website monitoring method.
As will be appreciated by one skilled in the art, embodiments of the present application may be provided as a method, system, or computer program product. Accordingly, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present application is described with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It will be apparent to those skilled in the art that various changes and modifications may be made in the present application without departing from the spirit and scope of the application. Thus, if such modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is intended to include such modifications and variations as well.
Claims (10)
1. A method for website monitoring, comprising:
acquiring a webpage of a website to be monitored; the webpage is obtained by crawling by a web crawler and/or recording by a search engine;
determining key indexes of all the webpages according to the crawling depth of the web crawler and/or the webpage updating time recorded by a search engine;
determining the criticality of each webpage according to the key index of each webpage, and sequencing; putting the sorted webpages into a webpage pyramid to be monitored with a preset number of layers; the number of the webpages put into each layer in the webpage pyramid to be monitored is determined by a preset index function;
determining the monitoring period of each layer in the pyramid of the web pages to be monitored and the number of the monitoring web pages corresponding to the monitoring period of each layer according to a preset reference monitoring period;
monitoring the webpages in each layer in the webpage pyramid to be monitored in a sliding mode according to the monitoring periods of each layer in the webpage pyramid to be monitored and the number of the monitored webpages corresponding to the monitoring periods of each layer.
2. The method of claim 1, wherein determining the key index of each web page according to the crawling depth of the web crawler and/or the update time of the web pages included in the search engine comprises:
determining the relation between a father page and a son page in each webpage according to the crawling depth of the breadth-first method of the web crawler; determining a first key index of each webpage according to the relationship between a father page and a son page in each webpage; the sum of the first key index of the parent page and a first preset value is the first key index of the child page; and/or
Obtaining the time difference between the webpage updating time of each webpage and the current time according to the webpage updating time recorded in the search engine of each webpage, and determining a second key index of each webpage according to the time difference;
and repeating the first key index of each webpage and the second key index of each webpage to obtain the key index of each webpage.
3. The method of claim 2, wherein the deduplicating the first key index of each web page and the second key index of each web page to obtain the key index of each web page comprises:
if the first key index and the second key index exist in the webpage, judging whether the first key index is larger than the second key index, if so, determining the first key index as the key index of the webpage, otherwise, determining the second key index as the key index of the webpage;
and if the first key index or the second key index exists in the webpage, determining the first key index or the second key index as the key index of the webpage.
4. The method of claim 1, wherein the determining the criticality of the web pages according to the criticality index of the web pages and ranking comprises:
determining the reciprocal of the key index of each webpage as the key degree of each webpage;
and sequencing the webpages according to the sequence of the criticality from large to small according to the criticality of the webpages.
5. The method of claim 1, wherein the determining the number of pages to be placed in each layer of the pyramid of pages to be monitored by a predetermined exponential function comprises:
determining a scale coefficient of the website according to the total number of the webpages of the website to be monitored and a second preset value number of the webpages which can be stored in the webpage pyramid to be monitored;
and determining the number of the webpages put into each layer in the pyramid of the webpages to be monitored according to the scale coefficient and a preset index function.
6. The method of claim 1, wherein the determining the monitoring period of each layer in the pyramid of web pages to be monitored and the number of monitoring web pages corresponding to the monitoring period of each layer comprises:
determining the monitoring period of each layer in the pyramid of the webpage to be monitored according to the preset reference monitoring period and the preset index function;
and determining the number of the monitoring webpages corresponding to the monitoring period of each layer according to the number of layers of the webpage pyramid to be monitored and the preset index function.
7. The method of claim 1, wherein the monitoring the webpages in each layer of the pyramid of the webpages to be monitored by sliding the monitoring webpages according to the monitoring periods of each layer in the pyramid of the webpages to be monitored and the number of the monitoring webpages corresponding to the monitoring periods of each layer comprises:
determining the number of the sliding webpages per monitoring period of each layer in the webpage pyramid to be monitored;
and selecting the webpages from each layer in the pyramid of the webpages to be monitored for monitoring according to the number of the webpages sliding in each monitoring period of each layer and the number of the monitoring webpages corresponding to the monitoring period of each layer.
8. An apparatus for website monitoring, comprising:
the acquisition module is used for acquiring a webpage of a website to be monitored; the webpage to be monitored is obtained by crawling by a web crawler and/or recording by a search engine;
the processing module is used for determining the key index of each webpage according to the crawling depth of the web crawler and/or the webpage updating time recorded by the search engine;
determining the criticality of each webpage according to the key index of each webpage, and sequencing; putting the sorted webpages into a webpage pyramid to be monitored with a preset number of layers; the number of the webpages put into each layer in the webpage pyramid to be monitored is determined by a preset index function;
determining the monitoring period of each layer in the pyramid of the web pages to be monitored and the number of the monitoring web pages corresponding to the monitoring period of each layer according to a preset reference monitoring period;
monitoring the webpages in each layer in the webpage pyramid to be monitored in a sliding mode according to the monitoring periods of each layer in the webpage pyramid to be monitored and the number of the monitored webpages corresponding to the monitoring periods of each layer.
9. A computing device, comprising:
a memory for storing program instructions;
a processor for calling program instructions stored in said memory to perform the method of any of claims 1 to 7 in accordance with the obtained program.
10. A computer-readable storage medium having stored thereon computer-executable instructions for causing a computer to perform the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010731482.3A CN111898013B (en) | 2020-07-27 | 2020-07-27 | Website monitoring method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010731482.3A CN111898013B (en) | 2020-07-27 | 2020-07-27 | Website monitoring method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111898013A true CN111898013A (en) | 2020-11-06 |
| CN111898013B CN111898013B (en) | 2024-03-19 |
Family
ID=73191000
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010731482.3A Active CN111898013B (en) | 2020-07-27 | 2020-07-27 | Website monitoring method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111898013B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113992378A (en) * | 2021-10-22 | 2022-01-28 | 绿盟科技集团股份有限公司 | Safety monitoring method and device, electronic equipment and storage medium |
Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006215735A (en) * | 2005-02-02 | 2006-08-17 | Mitsubishi Electric Corp | Duplicate website detection device |
| US20090077171A1 (en) * | 2007-09-17 | 2009-03-19 | Kasha John R | Systems and methods for monitoring webpages |
| CN101964026A (en) * | 2009-07-23 | 2011-02-02 | 中联绿盟信息技术(北京)有限公司 | Method and system for detecting web page horse hanging |
| US20130262427A1 (en) * | 2012-04-02 | 2013-10-03 | Microsoft Corporation | Context-sensitive deeplinks |
| WO2014190776A1 (en) * | 2013-05-28 | 2014-12-04 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for notifying a user of updated content for a webpage |
| US20140359490A1 (en) * | 2013-05-28 | 2014-12-04 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for notifying a user of updated content for a webpage |
| CN105528416A (en) * | 2015-12-07 | 2016-04-27 | 中南大学 | Method and system for monitoring update contents of website |
| CN105930513A (en) * | 2016-05-16 | 2016-09-07 | 北京京东尚科信息技术有限公司 | Browser history record sorting method and apparatus |
| CN106130810A (en) * | 2016-06-01 | 2016-11-16 | 北京百度网讯科技有限公司 | Website monitoring method and device |
| CN106209487A (en) * | 2015-05-07 | 2016-12-07 | 阿里巴巴集团控股有限公司 | For detecting the method and device of the security breaches of webpage in website |
| CN106778357A (en) * | 2016-12-23 | 2017-05-31 | 北京神州绿盟信息安全科技股份有限公司 | The detection method and device of a kind of webpage tamper |
| CN106874298A (en) * | 2015-12-14 | 2017-06-20 | 北京国双科技有限公司 | Page detection method and device |
| CN107301355A (en) * | 2017-06-20 | 2017-10-27 | 深信服科技股份有限公司 | A kind of webpage tamper monitoring method and device |
| CN107832428A (en) * | 2017-11-14 | 2018-03-23 | 北京知行锐景科技有限公司 | Webpage method for monitoring state and system based on Website page |
| WO2019152021A1 (en) * | 2018-01-31 | 2019-08-08 | Hewlett-Packard Development Company, L.P. | Monitoring web applications including microservices |
| US20190281064A1 (en) * | 2018-03-09 | 2019-09-12 | Microsoft Technology Licensing, Llc | System and method for restricting access to web resources |
| CN110851690A (en) * | 2019-11-14 | 2020-02-28 | 北京计算机技术及应用研究所 | Method and device for collecting network information of monitoring website |
-
2020
- 2020-07-27 CN CN202010731482.3A patent/CN111898013B/en active Active
Patent Citations (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2006215735A (en) * | 2005-02-02 | 2006-08-17 | Mitsubishi Electric Corp | Duplicate website detection device |
| US20090077171A1 (en) * | 2007-09-17 | 2009-03-19 | Kasha John R | Systems and methods for monitoring webpages |
| CN101964026A (en) * | 2009-07-23 | 2011-02-02 | 中联绿盟信息技术(北京)有限公司 | Method and system for detecting web page horse hanging |
| US20130262427A1 (en) * | 2012-04-02 | 2013-10-03 | Microsoft Corporation | Context-sensitive deeplinks |
| WO2014190776A1 (en) * | 2013-05-28 | 2014-12-04 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for notifying a user of updated content for a webpage |
| US20140359490A1 (en) * | 2013-05-28 | 2014-12-04 | Tencent Technology (Shenzhen) Company Limited | Method and apparatus for notifying a user of updated content for a webpage |
| CN106209487A (en) * | 2015-05-07 | 2016-12-07 | 阿里巴巴集团控股有限公司 | For detecting the method and device of the security breaches of webpage in website |
| CN105528416A (en) * | 2015-12-07 | 2016-04-27 | 中南大学 | Method and system for monitoring update contents of website |
| CN106874298A (en) * | 2015-12-14 | 2017-06-20 | 北京国双科技有限公司 | Page detection method and device |
| CN105930513A (en) * | 2016-05-16 | 2016-09-07 | 北京京东尚科信息技术有限公司 | Browser history record sorting method and apparatus |
| CN106130810A (en) * | 2016-06-01 | 2016-11-16 | 北京百度网讯科技有限公司 | Website monitoring method and device |
| CN106778357A (en) * | 2016-12-23 | 2017-05-31 | 北京神州绿盟信息安全科技股份有限公司 | The detection method and device of a kind of webpage tamper |
| CN107301355A (en) * | 2017-06-20 | 2017-10-27 | 深信服科技股份有限公司 | A kind of webpage tamper monitoring method and device |
| CN107832428A (en) * | 2017-11-14 | 2018-03-23 | 北京知行锐景科技有限公司 | Webpage method for monitoring state and system based on Website page |
| WO2019152021A1 (en) * | 2018-01-31 | 2019-08-08 | Hewlett-Packard Development Company, L.P. | Monitoring web applications including microservices |
| US20190281064A1 (en) * | 2018-03-09 | 2019-09-12 | Microsoft Technology Licensing, Llc | System and method for restricting access to web resources |
| CN110851690A (en) * | 2019-11-14 | 2020-02-28 | 北京计算机技术及应用研究所 | Method and device for collecting network information of monitoring website |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113992378A (en) * | 2021-10-22 | 2022-01-28 | 绿盟科技集团股份有限公司 | Safety monitoring method and device, electronic equipment and storage medium |
| CN113992378B (en) * | 2021-10-22 | 2023-11-07 | 绿盟科技集团股份有限公司 | Security monitoring method and device, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111898013B (en) | 2024-03-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9813450B1 (en) | Metadata-based verification of artifact quality policy compliance | |
| CN110321466B (en) | Securities information duplicate checking method and system based on semantic analysis | |
| US20160350379A1 (en) | Search results based on a search history | |
| CN103530365A (en) | Method and system for acquiring downloading link of resources | |
| WO2018095099A1 (en) | Method and device for processing suspicious samples | |
| CN112131507A (en) | Website content processing method, device, server and computer-readable storage medium | |
| CN107239563A (en) | Public feelings information dynamic monitoring and controlling method | |
| US20100106681A1 (en) | Identifying Files Associated With A Workflow | |
| CN109815697A (en) | Wrong report behavior processing method and processing device | |
| CN110363381B (en) | Information processing method and device | |
| CN111898013A (en) | Website monitoring method and device | |
| CN116226865A (en) | Security detection method, device, server, medium and product of cloud native application | |
| CN109446441B (en) | General credible distributed acquisition and storage system for network community | |
| CN110147473B (en) | Crawling method and device for crawler | |
| CN117725053A (en) | Alliance chain edge data perception predictor and construction method thereof | |
| CN111427871B (en) | Data processing method, device and equipment | |
| CN107729206A (en) | Real-time analysis method, system and the computer-processing equipment of alarm log | |
| CN108664322A (en) | Data processing method and system | |
| CN110991241A (en) | Abnormality recognition method, apparatus, and computer-readable medium | |
| CN110968779A (en) | Processing method and device for crawling webpage information | |
| CN115834124A (en) | Abnormal user detection method, device and computer program product | |
| CN106161542A (en) | A kind of data download method and device | |
| CN107451228B (en) | A kind of browser data backup method and device | |
| CN104462492B (en) | The method and apparatus for capturing question and answer class webpage | |
| CN111125482B (en) | Method and device for adjusting data crawling frequency, storage medium and processor |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |