CN116467504A - Crawler blocking method, equipment and storage medium - Google Patents
Crawler blocking method, equipment and storage medium Download PDFInfo
- Publication number
- CN116467504A CN116467504A CN202310350492.6A CN202310350492A CN116467504A CN 116467504 A CN116467504 A CN 116467504A CN 202310350492 A CN202310350492 A CN 202310350492A CN 116467504 A CN116467504 A CN 116467504A
- Authority
- CN
- China
- Prior art keywords
- crawler
- program
- browser
- instruction
- private
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 55
- 230000000903 blocking effect Effects 0.000 title claims abstract description 28
- 238000001514 detection method Methods 0.000 claims abstract description 24
- 238000012795 verification Methods 0.000 claims abstract description 14
- 231100000572 poisoning Toxicity 0.000 claims abstract description 10
- 230000000607 poisoning effect Effects 0.000 claims abstract description 10
- 238000012545 processing Methods 0.000 claims abstract description 9
- 238000012544 monitoring process Methods 0.000 claims abstract description 8
- 238000004422 calculation algorithm Methods 0.000 claims description 14
- 208000005374 Poisoning Diseases 0.000 claims description 9
- 230000003993 interaction Effects 0.000 claims description 9
- 230000009193 crawling Effects 0.000 abstract description 9
- 230000008569 process Effects 0.000 description 5
- 230000006870 function Effects 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000010276 construction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000011156 evaluation Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 239000000523 sample Substances 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 230000001131 transforming effect Effects 0.000 description 2
- PXFBZOLANLWPMH-UHFFFAOYSA-N 16-Epiaffinine Natural products C1C(C2=CC=CC=C2N2)=C2C(=O)CC2C(=CC)CN(C)C1C2CO PXFBZOLANLWPMH-UHFFFAOYSA-N 0.000 description 1
- 241000282326 Felis catus Species 0.000 description 1
- 206010033799 Paralysis Diseases 0.000 description 1
- 238000007373 indentation Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000011002 quantification Methods 0.000 description 1
- 230000001960 triggered effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/302—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system component is a software system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3058—Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/10—Protecting distributed programs or content, e.g. vending or licensing of copyrighted material ; Digital rights management [DRM]
- G06F21/12—Protecting executable software
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/602—Providing cryptographic facilities or services
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Computer Hardware Design (AREA)
- Computer Security & Cryptography (AREA)
- Quality & Reliability (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Technology Law (AREA)
- Health & Medical Sciences (AREA)
- Mathematical Physics (AREA)
- Storage Device Security (AREA)
Abstract
The invention relates to the technical field of computer software and discloses a crawler blocking method, equipment and a storage medium. The crawler blocking method comprises the following steps: when a program sends out a data request instruction, a plurality of detection parameters are obtained through monitoring the environment of the browser, and the detection parameters are combined to generate an encrypted character string; transmitting the encrypted character string to a server, and receiving a crawler judgment result of the server on the program; the server is used for decrypting the encrypted character string to obtain a decryption instruction, and judging whether the program is a crawler program or not through the decryption instruction; if the program is judged to be a crawler program, the program is subjected to black drawing processing or poisoning processing. The invention effectively prevents the violent crawling of the crawler program, maintains the smoothness of user access, and avoids introducing other complicated man-machine verification operations.
Description
Technical Field
The present invention relates to the field of computer software technologies, and in particular, to a crawler blocking method, device, and storage medium.
Background
Along with the coming of big data age, the value of data is increasingly greater, and the crawler is used as a way for acquiring internet data, and the application of the crawler is also increasingly wider. For a website, crawling of a crawler can effectively improve search engine optimization of the website and increase exposure of website content.
However, the crawling of the crawler also has some drawbacks, and particularly, the crawling of the crawler necessarily occupies certain resources, especially some malicious crawlers occupy a large amount of resources, however, the processing capacity of the website server and the resources such as network bandwidth are limited, so that on the premise of fixed total resources, the more the resources occupied by the crawler, the less the resources belonging to visitors, thus the service capacity of the website is reduced, and even the website is paralyzed; some other malicious crawlers may also attack the website.
Therefore, for the website, the crawling of the crawlers needs to be limited, and the crawling of the crawlers is limited, so that the crawlers are detected first.
The idea of the crawler detection is to summarize and generalize the access behaviors of the visitors and sort out a certain rule to judge whether the access behaviors are crawler accesses or not. The two common crawler detection methods at present are as follows: firstly, recording the ip address of a visitor and the access times of an ip address in a certain time, and if the access times exceed a certain threshold value, identifying the visitor as a crawler; secondly, adopting an authentication login mode to limit the access evaluation rate of the user, and carrying out identity authentication on the user with high access evaluation rate; the user is often authenticated for verification codes such as in a cat's anti-climb strategy,
for the first method for detecting the crawler, the crawler cannot be identified when the crawler actively controls the crawling frequency or frequently replaces ip to access; for the second method of crawler detection, some crawlers already exist to support the ability to disguise user identification verification codes, and therefore cannot be identified for such crawlers. To sum up, the existing crawler detection mode cannot more effectively identify and detect the crawler.
Disclosure of Invention
The invention provides a crawler blocking method, equipment and a storage medium, which are used for solving the technical problem of how to detect a crawler program when a browser interacts with server data in the prior art.
The first aspect of the invention provides a crawler blocking method, which comprises the following steps:
when a program sends out a data request instruction, a plurality of detection parameters are obtained through monitoring the environment of the browser, and the detection parameters are combined to generate an encrypted character string;
transmitting the encrypted character string to a server, and receiving a crawler judgment result of the server on the program; the server is used for decrypting the encrypted character string to obtain a decryption instruction, and judging whether the program is a crawler program or not through the decryption instruction;
if the program is judged to be a crawler program, the program is subjected to black drawing processing or poisoning processing.
Optionally, in a first implementation manner of the first aspect of the present invention, the method further includes:
setting one or more cascading style sheets in an html file in a browser, wherein the one or more cascading style sheets are used for identifying a crawler program;
an abstract syntax tree is used for html files in a browser and used for dynamically confusing data;
an encryption algorithm is configured in the browser and server data interaction for encrypting the exchanged data.
Optionally, in a second implementation manner of the first aspect of the present invention, the method includes:
receiving signals of clicking and moving of a mouse and signals input by a keyboard, and judging verification conditions of the movement track of the mouse and the signals input by the keyboard; if the movement track of the mouse and the signals input by the keyboard do not accord with the verification conditions, judging that the program is a crawler program, and carrying out black drawing treatment or poisoning treatment on the program.
Optionally, in a third implementation manner of the first aspect of the present invention, the using an encryption algorithm in the browser and server data interaction includes: the browser script is virtualized encrypted by JavaScript Virtual Machine Protection.
Optionally, in a fourth implementation manner of the first aspect of the present invention, the using an encryption algorithm in the browser and server data interaction includes:
the code written in the programming language is converted into mechanical code recognized by the browser through wasm, or encrypted through a block encryption algorithm.
Optionally, in a fifth implementation manner of the first aspect of the present invention, the step of JavaScript Virtual Machine Protection virtualizing and encrypting includes:
server side:
reading js codes;
analyzing whether the grammar of js code is correct or not;
if the private instruction is correct, compiling and generating the private instruction, encrypting the private instruction, and generating a corresponding private interpretation;
sending the private interpretation to a browser;
browser end:
loading a private interpreter;
decrypting the private interpretation based on the private interpreter to obtain a private instruction;
explaining the private instruction to obtain an execution instruction;
judging whether the execution of the execution instruction is completed, ending if the execution of the execution instruction is completed, and returning to the step of decrypting the private interpretation if the execution of the execution instruction is not completed.
Optionally, in a sixth implementation manner of the first aspect of the present invention, the method includes: and setting quantitative display pages for displaying the set page data in the html file of the browser, and placing the rest data which are not displayed on client software for display.
Optionally, in a seventh implementation manner of the first aspect of the present invention, the plurality of detection parameters includes: webdriver parameters, module parameters, etc.
A second aspect of the present invention provides a crawler blocking apparatus comprising: a memory and at least one processor, the memory having instructions stored therein;
the at least one processor invokes the instructions in the memory to cause the crawler blocking device to perform the crawler blocking method as described above.
A third aspect of the present invention provides a computer readable storage medium having instructions stored thereon which when executed by a processor implement a crawler blocking method as described above.
In the technical scheme provided by the invention, the beneficial effects are as follows: according to the crawler blocking method, the crawler blocking equipment and the storage medium, whether the program is a crawler program is judged by monitoring the environment of the browser, and if the program is judged to be the crawler program, the program is subjected to black drawing treatment or poisoning treatment. The invention greatly reduces the frequency of crawler collection, accelerates the response speed of the website, effectively prevents the crawlers from crawling data, and achieves the effect of shielding most crawlers without affecting the experience of normal users in access.
Drawings
FIG. 1 is a schematic diagram of an embodiment of a crawler blocking method according to an embodiment of the present invention.
Detailed Description
The embodiment of the invention provides a crawler blocking method, equipment and a storage medium, which are used for solving the technical problem of how to detect a crawler program when a browser interacts with server data in the prior art.
The terms "first," "second," "third," "fourth" and the like in the description and in the claims and in the above drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments described herein may be implemented in other sequences than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
For ease of understanding, the following describes a specific flow of an embodiment of the present invention, and fig. 1 is a schematic diagram of an embodiment of a crawler blocking method in an embodiment of the present invention, referring to fig. 1, an embodiment of a crawler blocking method in an embodiment of the present invention includes:
step 101: when a program sends out a data request instruction, a plurality of detection parameters are obtained through monitoring the environment of the browser, and the detection parameters are combined to generate an encrypted character string;
step 102: transmitting the encrypted character string to a server, and receiving a crawler judgment result of the server on the program; the server is used for decrypting the encrypted character string to obtain a decryption instruction, and judging whether the program is a crawler program or not through the decryption instruction;
step 103: if the program is judged to be a crawler program, the program is subjected to black drawing processing or poisoning processing.
In particular, many crawlers are simple simulated browser environments such as User-Agent (User-Agent, a special string header.) that detect the presence of a crawlers by adding the environment monitoring code of the browser to jsvmp.
The browser environment monitoring method comprises the following steps:
acquiring webdriver parameters by detecting whether the browser is an automatic program; if so, generating a detection parameter, and setting the detection parameter to True, otherwise, setting the detection parameter to False.
Specifically, when developing a webpage, a developer sets an event through JavaScript, and then invokes a WebDriver attribute of a Navigator object through the event, so as to judge whether the client uses the WebDriver to drive a browser.
If the webdriver attribute is detected to exist in the client, returning True, and judging that the client is a crawler program at the moment; if not, return False.
Detecting differences between the browser and the node. Js environment (the common running environment of the crawler), for example, detecting whether a module parameter exists, and if not, the normal browser, the encryption generated by the crawler is mostly executed in the node. Js environment, and the parameter exists; if a module parameter is present, a detection parameter is generated and set to True, otherwise the detection parameter is set to False.
Detecting whether the Jsvmp code is formatted; the Jsvmp code is formatted to be more standardized, necessary line wrapping and indentation are reserved to enable the crawler program to be unrecognizable, if the Jsvmp code is formatted, the detection parameter is generated, the detection parameter is set to True, and otherwise, the detection parameter is set to False.
Other probe codes, like probe programs, such as del window and del navigator, are added to the obfuscated code, and because they cannot be deleted in the browser, but can be deleted in nodejs, the difference is used to determine whether the parameters are generated in the browser or nodejs, if they cannot be deleted, the parameters are generated, and the parameters are set to true, otherwise they are false.
And combining the detection parameters to generate an encryption character string, transmitting the encryption character string to a server, decrypting the encryption character string by the server to obtain a decryption instruction, judging that the program is a crawler program if the decryption instruction represents True meaning, and judging that the program is not the crawler program if the decryption instruction represents False meaning.
And the browser receives a crawler judgment result of the server on the program, and if the program is judged to be the crawler program, the program is subjected to blackening treatment or poisoning treatment.
In an embodiment, the method further comprises:
setting one or more cascading style sheets in an html file in a browser, wherein the one or more cascading style sheets are used for identifying a crawler program;
specifically, a cascading style sheet (English full name: cascading Style Sheets) is a computer language for representing file styles such as HTML (an application of standard general markup language). The web page is statically decorated with CSS, and the elements of the web page are formatted dynamically in coordination with various scripting languages. The CSS can accurately control the typesetting of element positions in the webpage at a pixel level, a probe program is embedded in an html file of a browser (one or more CSS in the html file are set to be not displayed, a normal user cannot click, a crawler program is triggered by going through the crawler program), and once a hidden link is accessed, the hidden link is identified as the crawler program.
The method further comprises the steps of: an abstract syntax tree is used for html files in a browser and used for dynamically confusing data; the method comprises the following specific steps:
lexical Analysis (Lexical Analysis): reading each character in source code, converting it into token, and converting source code into list of token.
Syntax Analysis (syncax Analysis): converting list of token analyzed by the lexical analyzer into tree representation (tree expression);
finally, by lexical analysis and grammatical analysis, our code is converted into a tree node, all of which are combined to form concrete syntax tree (hybrid grammar tree)
The dynamic confusion data is realized by parsing (mapping) a source code, converting a code character string into an AST abstract syntax tree, translating (transforming), transforming the abstract syntax tree, and generating (generating) a new code character string according to the transformed abstract syntax tree.
The method further comprises the steps of: an encryption algorithm is configured in the browser and server data interaction for encrypting the exchanged data.
In an embodiment, the method further comprises: receiving signals of clicking and moving of a mouse and signals input by a keyboard, and judging verification conditions of the movement track of the mouse and the signals input by the keyboard; if the movement track of the mouse and the signals input by the keyboard do not accord with the verification conditions, judging that the program is a crawler program, and carrying out black drawing treatment or poisoning treatment on the program.
When the crawler accesses the browser, human-computer verification is needed, the browser judges whether the crawler crawls information on the browser according to whether the movement track of the mouse meets the preset verification condition or not when the browser receives the signal of clicking and moving the mouse, the situation is mostly judged by adding a logic picture verification code, when the browser receives the signal input by the keyboard, whether the signal input by the keyboard meets the preset verification condition or not and judges whether the information is crawled by the crawler on the browser or not, and the situation is mostly judged by a digital or logic character verification mode.
In one embodiment, the using an encryption algorithm in browser-to-server data interactions includes: the browser script is virtualized encrypted by JavaScript Virtual Machine Protection. Specifically, the method comprises the following steps:
step 1, dividing javascript codes to be protected into different code segments, and dividing the code segments into attributes, method types and computation intensive types according to the existence of dom attributes in the code segments;
step 2, generating corresponding interpreters of the attribute and the functions realized by the method type code segment and the computation intensive code segment respectively;
step 3, splitting an abstract syntax tree of the javascript code to be protected to obtain an intermediate code; generating and encoding a corresponding virtual instruction of the interpreter handler, and representing the intermediate code by using the virtual instruction so as to obtain a virtualized code;
and 4, respectively establishing and compiling a virtual interpreter of the computationally intensive code segment and a virtual interpreter of the attribute and method type code segment.
In one embodiment, the using an encryption algorithm in browser-to-server data interactions includes:
the code written in the programming language is converted into mechanical code recognized by the browser through wasm, or encrypted through a block encryption algorithm.
Specifically, the block encryption algorithm includes AES encryption.
Briefly, the encryption process of AES is broken down into four steps:
1. the plaintext is grouped into groups of 16 bytes (128 bits) and split into blocks of plaintext.
2. The proper padding mode is selected to fill the last plaintext block, ensuring that all plaintext blocks are 128 bits in length.
3. Each plaintext block is encrypted into a ciphertext block using an AES encryptor and a key.
4. And splicing all the ciphertext blocks to obtain a final ciphertext result.
The AES encryption comprises the following specific steps:
mapping from one byte to another is accomplished by the S-box,
wherein S-box construction c of sms4 implements the construction of S-boxes in the sms4 algorithm as a complex of three functions, the formula S (x) =l (I (L (x))), where L (x) is an affine transformation and I (x) is an inverse transformation;
the function of the row shift is to implement a permutation between bytes within a 4x4 matrix, the operation of the shift being: the first line is kept unchanged, the second line is circularly shifted left by 1 byte, the third line is circularly shifted left by 2 bytes, and the fourth line is circularly shifted left by 3 bytes;
from the multiplication of the matrix, it is known that in the process of column aliasing (using one substitution of the arithmetic property over the field GF (28)), the value corresponding to each byte has only a relation to the 4 values of the column, where GF (28) is GF (2^8), and 8 is just the number of bits of one byte;
in the encryption process, the input of each round is xored with the round key once (the current packet and a part of the expansion key are xored by bits); since the binary number is continuously exclusive-ored with a number, the key of the round is exclusive-ored again at decryption to recover the input.
In one embodiment, the JavaScript Virtual Machine Protection virtualized encryption step includes:
server side:
reading js codes;
analyzing whether the grammar of js code is correct or not;
if the private instruction is correct, compiling and generating the private instruction, encrypting the private instruction, and generating a corresponding private interpretation;
sending the private interpretation to a browser;
browser end:
loading a private interpreter;
decrypting the private interpretation based on the private interpreter to obtain a private instruction;
explaining the private instruction to obtain an execution instruction;
judging whether the execution of the execution instruction is completed, ending if the execution of the execution instruction is completed, and returning to the step of decrypting the private interpretation if the execution of the execution instruction is not completed.
In an embodiment, further comprising: and setting quantitative display pages for displaying the set page data in the html file of the browser, and placing the rest data which are not displayed on client software for display.
Wherein, the display page for setting the quantification includes, but is not limited to, 10 pages and 5 pages. The pages which are not displayed comprise, but are not limited to 25 pages and 20 pages, the client software comprises a mobile terminal APP, and the client software is used for storing other page data which are not displayed.
In an embodiment, the plurality of detection parameters includes: webdriver parameters, module parameters, etc.
The present invention also provides a crawler blocking apparatus including: a memory and at least one processor, the memory having instructions stored therein;
the at least one processor invokes the instructions in the memory to cause the crawler blocking device to perform the crawler blocking method described above.
The present invention also provides a computer readable storage medium having instructions stored thereon that when executed by a processor implement a crawler blocking method as described above.
In the embodiment of the invention, the beneficial effects are as follows: according to the crawler blocking method, the crawler blocking equipment and the storage medium, whether the program is a crawler program is judged by monitoring the environment of the browser, and if the program is judged to be the crawler program, the program is subjected to black drawing treatment or poisoning treatment. The invention greatly reduces the frequency of crawler collection, accelerates the response speed of the website, effectively prevents the crawlers from crawling data, and achieves the effect of shielding most crawlers without affecting the experience of normal users in access.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The above embodiments are only for illustrating the technical solution of the present invention, and not for limiting the same; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (10)
1. A crawler blocking method, the method being applied to a browser, comprising:
when a program sends out a data request instruction, a plurality of detection parameters are obtained through monitoring the environment of the browser, and the detection parameters are combined to generate an encrypted character string;
transmitting the encrypted character string to a server, and receiving a crawler judgment result of the server on the program; the server is used for decrypting the encrypted character string to obtain a decryption instruction, and judging whether the program is a crawler program or not through the decryption instruction;
if the program is judged to be a crawler program, the program is subjected to black drawing processing or poisoning processing.
2. The method according to claim 1, wherein the method further comprises:
setting one or more cascading style sheets in an html file in a browser, wherein the one or more cascading style sheets are used for identifying a crawler program;
an abstract syntax tree is used for html files in a browser and used for dynamically confusing data;
an encryption algorithm is configured in the browser and server data interaction for encrypting the exchanged data.
3. The method according to claim 1, characterized in that it comprises:
receiving signals of clicking and moving of a mouse and signals input by a keyboard, and judging verification conditions of the movement track of the mouse and the signals input by the keyboard; if the movement track of the mouse and the signals input by the keyboard do not accord with the verification conditions, judging that the program is a crawler program, and carrying out black drawing treatment or poisoning treatment on the program.
4. The method of claim 1, wherein using an encryption algorithm in browser-to-server data interactions comprises: the browser script is virtualized encrypted by JavaScript Virtual Machine Protection.
5. The method of claim 1, wherein using an encryption algorithm in browser-to-server data interactions comprises:
the code written in the programming language is converted into mechanical code recognized by the browser through wasm, or encrypted through a block encryption algorithm.
6. The method of claim 4, wherein the JavaScript Virtual Machine Protection virtualized encryption step comprises:
server side:
reading js codes;
analyzing whether the grammar of js code is correct or not;
if the private instruction is correct, compiling and generating the private instruction, encrypting the private instruction, and generating a corresponding private interpretation;
sending the private interpretation to a browser;
browser end:
loading a private interpreter;
decrypting the private interpretation based on the private interpreter to obtain a private instruction;
explaining the private instruction to obtain an execution instruction;
judging whether the execution of the execution instruction is completed, ending if the execution of the execution instruction is completed, and returning to the step of decrypting the private interpretation if the execution of the execution instruction is not completed.
7. The method as recited in claim 2, further comprising: and setting quantitative display pages for displaying the set page data in the html file of the browser, and placing the rest data which are not displayed on client software for display.
8. The method of claim 1, wherein the plurality of detection parameters comprises: webdriver parameters, module parameters, etc.
9. A crawler blocking apparatus, characterized in that it comprises: a memory and at least one processor, the memory having instructions stored therein;
the at least one processor invoking the instructions in the memory to cause the crawler blocking device to perform the crawler blocking method of any of claims 1-8.
10. A computer readable storage medium having instructions stored thereon, which when executed by a processor implement the crawler blocking method of any of claims 1-8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310350492.6A CN116467504A (en) | 2023-03-28 | 2023-03-28 | Crawler blocking method, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310350492.6A CN116467504A (en) | 2023-03-28 | 2023-03-28 | Crawler blocking method, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116467504A true CN116467504A (en) | 2023-07-21 |
Family
ID=87174491
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310350492.6A Pending CN116467504A (en) | 2023-03-28 | 2023-03-28 | Crawler blocking method, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116467504A (en) |
-
2023
- 2023-03-28 CN CN202310350492.6A patent/CN116467504A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US10567407B2 (en) | Method and system for detecting malicious web addresses | |
| US9712560B2 (en) | Web page and web browser protection against malicious injections | |
| US9836612B2 (en) | Protecting data | |
| US8543869B2 (en) | Method and system for reconstructing error response messages under web application environment | |
| US9553865B2 (en) | Protecting websites from cross-site scripting | |
| US20100332993A1 (en) | Method and system for delivering digital content | |
| Jonker et al. | Fingerprint surface-based detection of web bot detectors | |
| WO2016113663A1 (en) | Rasp for scripting languages | |
| KR101190261B1 (en) | Hybrid interaction client honeypot system and its operation method | |
| US11586726B2 (en) | Secure web framework | |
| CN111212033A (en) | Page display method and device based on combined web crawler defense technology and electronic equipment | |
| CN112989348A (en) | Attack detection method, model training method, device, server and storage medium | |
| CN108319822B (en) | Method, storage medium, electronic device and system for protecting webpage code | |
| Alnaeli et al. | Source code vulnerabilities in IoT software systems | |
| CN109684844B (en) | Webshell detection method and device, computing equipment and computer-readable storage medium | |
| CN116467504A (en) | Crawler blocking method, equipment and storage medium | |
| CN109218284B (en) | XSS vulnerability detection method and device, computer equipment and readable medium | |
| US20220253510A1 (en) | Web Browser Extension Script Obfuscation System | |
| WO2023031679A1 (en) | Systems and methods for inhibiting exploitations in runtime environments | |
| RU2702081C2 (en) | Web property modification detection system and method | |
| CN111090856A (en) | Crawler detection method based on browser feature detection and event monitoring | |
| Ashouri | A Large-Scale Analysis of Browser Fingerprinting via Chrome Instrumentation | |
| CN110661759B (en) | Access detection method and device | |
| JP7013297B2 (en) | Fraud detection device, fraud detection network system, and fraud detection method | |
| KR102311119B1 (en) | Method for automatic diagnosis vulnerability of web and apparatus for performing the method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |