CN113609501A - Anti-crawler method and system based on asymmetric secondary encryption - Google Patents
Anti-crawler method and system based on asymmetric secondary encryption Download PDFInfo
- Publication number
- CN113609501A CN113609501A CN202110895961.3A CN202110895961A CN113609501A CN 113609501 A CN113609501 A CN 113609501A CN 202110895961 A CN202110895961 A CN 202110895961A CN 113609501 A CN113609501 A CN 113609501A
- Authority
- CN
- China
- Prior art keywords
- user
- crawler
- encryption
- client
- asymmetric
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 24
- 238000012795 verification Methods 0.000 claims abstract description 19
- 235000014510 cooky Nutrition 0.000 claims description 9
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000007405 data analysis Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/604—Tools and structures for managing or administering access control systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/31—User authentication
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
- G06F21/45—Structures or tools for the administration of authentication
- G06F21/46—Structures or tools for the administration of authentication by designing passwords or checking the strength of passwords
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
- G06F21/60—Protecting data
- G06F21/62—Protecting access to data via a platform, e.g. using keys or access control rules
- G06F21/6218—Protecting access to data via a platform, e.g. using keys or access control rules to a system of files or objects, e.g. local or distributed file system or database
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/54—Interprogram communication
- G06F9/547—Remote procedure calls [RPC]; Web services
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
- H04L63/0442—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload wherein the sending and receiving network entities apply asymmetric encryption, i.e. different keys for encryption and decryption
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L63/00—Network architectures or network communication protocols for network security
- H04L63/04—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks
- H04L63/0428—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload
- H04L63/0478—Network architectures or network communication protocols for network security for providing a confidential data exchange among entities communicating through data packet networks wherein the data content is protected, e.g. by encrypting or encapsulating the payload applying multiple layers of encryption, e.g. nested tunnels or encrypting the content with a first key and then with at least a second key
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/54—Indexing scheme relating to G06F9/54
- G06F2209/549—Remote execution
Landscapes
- Engineering & Computer Science (AREA)
- Computer Security & Cryptography (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Computer Hardware Design (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Health & Medical Sciences (AREA)
- Bioethics (AREA)
- General Health & Medical Sciences (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computing Systems (AREA)
- Automation & Control Theory (AREA)
- Databases & Information Systems (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses an anti-crawler method and system based on asymmetric secondary encryption, wherein the method comprises the following steps: verifying user login; encrypting once based on the client to generate sign 01; responding to user access, calling RPC micro-service verification sign01, and performing secondary encryption after verification is passed to generate sign 02; and the user accesses the target server based on sign02, the target server successfully verifies and returns data. The system comprises: client, user side and server side. By using the invention, the purpose of anti-crawler is realized, and the information safety is ensured. The crawler-resisting method and system based on asymmetric secondary encryption can be widely applied to the field of network data security protection.
Description
Technical Field
The invention relates to the field of network data security protection, in particular to an anti-crawler method and system based on asymmetric secondary encryption.
Background
There are countless web pages on the internet, and the web pages store massive information, and if these data can be obtained for data analysis, huge commercial value will be generated. Therefore, a crawler is generated, and the program can be used for automatically acquiring webpage content, automatically analyzing the webpage and extracting corresponding content of data. The existing signature verification technical scheme has the defects that the signature is generated at the front end of the browser, and a crawler engineer can find a function method for generating the signature by looking up Javascript codes in the browser, so as to forge the signature and crack the anti-crawler.
Disclosure of Invention
In order to solve the above technical problems, an object of the present invention is to provide an anti-crawler method and system based on asymmetric twice encryption, which places an encryption function of the second encryption at a server side, so that a crawler engineer cannot crack a generation rule of an encryption parameter, thereby protecting data.
The first technical scheme adopted by the invention is as follows: an anti-crawler method based on asymmetric twice encryption comprises the following steps:
verifying user login;
encrypting once based on the client to generate sign 01;
responding to user access, calling RPC micro-service verification sign01, and performing secondary encryption after verification is passed to generate sign 02;
and the user accesses the target server based on sign02, the target server successfully verifies and returns data.
Further, the step of verifying the user login further includes:
a user submits an account password to log in and writes a cookie value to a client;
cookies values were recorded.
Further, the step of generating sign01 by performing encryption once based on the client specifically includes:
and judging that the user logs in successfully, and generating a code by the client through javascript based on a preset rule to obtain a sign01 parameter.
Further, the second encryption is performed after the verification is passed, so as to generate sign02, which specifically includes:
sequencing the service request parameters according to the alphabetical order;
linking the parameter name and the parameter value into a character string A;
adding a key at the tail of the character string A to form a new character string B;
the secret key is stored in the server side;
and performing MD5 hash operation on the character string to obtain an API signature sign and constructing sign 02.
The second technical scheme adopted by the invention is as follows: an anti-crawler system based on asymmetric quadratic encryption, comprising:
the client is used for verifying user login, recording Cookies values, generating sign01 and calling RPC micro-service;
the user side is used for submitting the user account password, accessing the server and receiving the returned data;
and the server side is used for verifying sign02 and returning data after verification is passed.
The method and the system have the beneficial effects that: according to the method, the user is preliminarily filtered through login verification, illegal requests are rejected, the difficulty of the crawler is greatly increased by adopting a secondary encryption technology, and further, an encryption function of secondary encryption is placed at a server end, so that a crawler engineer cannot crack the generation rule of encryption parameters, and data are protected.
Drawings
FIG. 1 is a flow chart of steps of an anti-crawler method based on asymmetric twice encryption of the present invention;
FIG. 2 is a schematic flow chart of an embodiment of the present invention.
Detailed Description
The invention is described in further detail below with reference to the figures and the specific embodiments. The step numbers in the following embodiments are provided only for convenience of illustration, the order between the steps is not limited at all, and the execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.
Referring to fig. 1 and 2, the invention provides an anti-crawler method based on asymmetric twice encryption, which deploys a micro-service to provide RPC remote call to generate sign signature function, and comprises the following steps:
verifying user login;
firstly, a user needs to perform login operation in the process of browsing a webpage, wherein the login operation is user filtering, namely, illegal users are denied access.
Encrypting once based on the client to generate sign 01;
responding to user access, calling RPC micro-service verification sign01, and performing secondary encryption after verification is passed to generate sign 02;
specifically, after login is finished, a user wants to access a link, needs to access the microservice, needs to perform parameter verification once, generates a code by javascript, and allows to call the RPC to generate a sign parameter under the condition that the parameter verification is passed. Since the sign encryption parameters for the second time are provided by RPC, a crawler engineer cannot know how the server generates the encryption parameters, so that the parameters cannot be cracked.
And the user accesses the target server based on sign02, the target server successfully verifies and returns data.
Specifically, after obtaining sign parameters provided by the RPC server, the user accesses the target server, and the target server can access the page and obtain data after verification is passed
Further as a preferred embodiment of the method, the step of verifying the user login further includes:
a user submits an account password to log in and writes a cookie value to a client;
cookies values were recorded.
Further, as a preferred embodiment of the method, the step of generating the sign01 by performing encryption once based on the client specifically includes:
and judging that the user logs in successfully, and generating a code by the client through javascript based on a preset rule to obtain a sign01 parameter.
Further, as a preferred embodiment of the method, performing secondary encryption after the verification passes to generate sign02, which specifically includes:
sequencing the service request parameters according to the alphabetical order;
linking the parameter name and the parameter value into a character string A;
adding a key appsecret to the tail of the character string A to form a new character string B;
for example: appsecret ═ dfhladkfhalkdhfa; dsdkfjhask;
the secret key is stored in the server side;
and performing MD5 hash operation on the character string to obtain an API signature sign and constructing sign 02.
Assume that the requested parameters are: f is 1, b is 23, k is 33, b is 23, f is 1, k is 33, b23f1k33 is obtained after the parameter name and the parameter value are linked, and b23f1k33 dfhladkfhalkdsfa is obtained after appsect is added to the tail; dsdkfjhask;
md5 then generates a signature sign, which is md5(b23f1k33 dfhladkfhalkhakdshfa; dsdkfjhask).
And when the server side verifies, verifying according to the sequence.
An anti-crawler system based on asymmetric quadratic encryption, comprising:
the client is used for verifying user login, recording Cookies values, generating sign01 and calling RPC micro-service;
the user side is used for submitting the user account password, accessing the server and receiving the returned data;
and the server side is used for verifying sign02 and returning data after verification is passed.
The contents in the above method embodiments are all applicable to the present system embodiment, the functions specifically implemented by the present system embodiment are the same as those in the above method embodiment, and the beneficial effects achieved by the present system embodiment are also the same as those achieved by the above method embodiment.
While the preferred embodiments of the present invention have been illustrated and described, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (5)
1. An anti-crawler method based on asymmetric twice encryption is characterized by comprising the following steps:
verifying user login;
encrypting once based on the client to generate sign 01;
responding to user access, calling RPC micro-service verification sign01, and performing secondary encryption after verification is passed to generate sign 02;
and the user accesses the target server based on sign02, the target server successfully verifies and returns data.
2. The asymmetric twice encryption-based crawler resisting method as recited in claim 1, wherein the step of verifying user login further comprises:
a user submits an account password to log in and writes a cookie value to a client;
cookies values were recorded.
3. The asymmetric twice encryption-based crawler resisting method as claimed in claim 2, wherein said step of generating sign01 by performing once encryption based on the client specifically comprises:
and judging that the user logs in successfully, and generating a code by the client through javascript based on a preset rule to obtain a sign01 parameter.
4. The asymmetric twice encryption-based anti-crawler method according to claim 3, wherein the twice encryption is performed after the verification is passed, and a sign02 is generated, which specifically includes:
sequencing the service request parameters according to the alphabetical order;
linking the parameter name and the parameter value into a character string A;
adding a key at the tail of the character string A to form a new character string B;
the secret key is stored in the server side;
and performing MD5 hash operation on the character string to obtain an API signature sign and constructing sign 02.
5. An anti-crawler system based on asymmetric twice encryption, comprising:
the client is used for verifying user login, recording Cookies values, generating sign01 and calling RPC micro-service;
the user side is used for submitting the user account password, accessing the server and receiving the returned data;
and the server side is used for verifying sign02 and returning data after verification is passed.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110895961.3A CN113609501B (en) | 2021-08-05 | 2021-08-05 | Anti-crawler method and system based on asymmetric secondary encryption |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110895961.3A CN113609501B (en) | 2021-08-05 | 2021-08-05 | Anti-crawler method and system based on asymmetric secondary encryption |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113609501A true CN113609501A (en) | 2021-11-05 |
| CN113609501B CN113609501B (en) | 2024-07-02 |
Family
ID=78307039
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110895961.3A Active CN113609501B (en) | 2021-08-05 | 2021-08-05 | Anti-crawler method and system based on asymmetric secondary encryption |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113609501B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140245228A1 (en) * | 2007-08-22 | 2014-08-28 | 9224-5489 Quebec Inc. | Method and apparatus for identifying user-selectable elements having a commonality thereof |
| CN107770171A (en) * | 2017-10-18 | 2018-03-06 | 厦门集微科技有限公司 | The verification method and system of the anti-reptile of server |
| CN108259479A (en) * | 2017-12-28 | 2018-07-06 | 广州启生信息技术有限公司 | Business data processing method, client and computer readable storage medium |
| CN111817845A (en) * | 2019-04-11 | 2020-10-23 | 亿度慧达教育科技（北京）有限公司 | Anti-crawler method and computer storage medium |
| CN112165475A (en) * | 2020-09-22 | 2021-01-01 | 成都知道创宇信息技术有限公司 | Anti-crawler method, anti-crawler device, website server and readable storage medium |
| CN112688919A (en) * | 2020-12-11 | 2021-04-20 | 杭州安恒信息技术股份有限公司 | APP interface-based crawler-resisting method, device and medium |
| CN112804269A (en) * | 2021-04-14 | 2021-05-14 | 中建电子商务有限责任公司 | Method for realizing website interface anti-crawler |
-
2021
- 2021-08-05 CN CN202110895961.3A patent/CN113609501B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20140245228A1 (en) * | 2007-08-22 | 2014-08-28 | 9224-5489 Quebec Inc. | Method and apparatus for identifying user-selectable elements having a commonality thereof |
| CN107770171A (en) * | 2017-10-18 | 2018-03-06 | 厦门集微科技有限公司 | The verification method and system of the anti-reptile of server |
| CN108259479A (en) * | 2017-12-28 | 2018-07-06 | 广州启生信息技术有限公司 | Business data processing method, client and computer readable storage medium |
| CN111817845A (en) * | 2019-04-11 | 2020-10-23 | 亿度慧达教育科技（北京）有限公司 | Anti-crawler method and computer storage medium |
| CN112165475A (en) * | 2020-09-22 | 2021-01-01 | 成都知道创宇信息技术有限公司 | Anti-crawler method, anti-crawler device, website server and readable storage medium |
| CN112688919A (en) * | 2020-12-11 | 2021-04-20 | 杭州安恒信息技术股份有限公司 | APP interface-based crawler-resisting method, device and medium |
| CN112804269A (en) * | 2021-04-14 | 2021-05-14 | 中建电子商务有限责任公司 | Method for realizing website interface anti-crawler |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113609501B (en) | 2024-07-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107426181B (en) | The hold-up interception method and device of malice web access request | |
| US9032085B1 (en) | Identifying use of software applications | |
| CN102073822A (en) | Method and system for preventing user information from leaking | |
| CN106549909B (en) | Authorization verification method and device | |
| CN109726578B (en) | Dynamic two-dimensional code anti-counterfeiting solution | |
| CN113315637A (en) | Security authentication method, device and storage medium | |
| CN112182614A (en) | Dynamic Web application protection system | |
| CN110071937A (en) | Login method, system and storage medium based on block chain | |
| CN112118238A (en) | Method, device, system, equipment and storage medium for authentication login | |
| CN116324766A (en) | Optimizing crawling requests by browsing profiles | |
| CN112149068A (en) | Access-based authorization verification method, information generation method and device, and server | |
| CN117040804A (en) | Network attack detection method, device, equipment, medium and program product for website | |
| US20230057802A1 (en) | System, method and architecture for secure sharing of customer intelligence | |
| CN113609501B (en) | Anti-crawler method and system based on asymmetric secondary encryption | |
| Ghiani et al. | Security in migratory interactive web applications | |
| CN111368231B (en) | Method and device for testing heterogeneous redundancy architecture website | |
| CN110971606B (en) | Construction method and application method of HACCP (Hadoop distributed control protocol) security system in Web application development | |
| Pevnev et al. | Web application protection technologies | |
| Riesch et al. | Audit based privacy preservation for the OpenID authentication protocol | |
| WO2018166365A1 (en) | Method and device for recording website access log | |
| CN111931159A (en) | Method and system for verifying validity of webpage data interface | |
| Mingsheng et al. | Research and Development of Dual-Core Browser-Based Compatibility and Security | |
| Srivastava et al. | Security and Scalability of E-Commerce Website by OWASP threats. | |
| Castillo et al. | Prevention of SQL injection attacks to login page of a website application using prepared statement technique | |
| Usman et al. | Building Secure Web-Applications Using Threat Model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |