CN112800311A - Browser page data acquisition method, terminal device and storage medium - Google Patents
Browser page data acquisition method, terminal device and storage medium Download PDFInfo
- Publication number
- CN112800311A CN112800311A CN202110158353.4A CN202110158353A CN112800311A CN 112800311 A CN112800311 A CN 112800311A CN 202110158353 A CN202110158353 A CN 202110158353A CN 112800311 A CN112800311 A CN 112800311A
- Authority
- CN
- China
- Prior art keywords
- page
- browser
- data acquisition
- task
- access
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 38
- 238000012545 processing Methods 0.000 claims abstract description 16
- 238000004590 computer program Methods 0.000 claims description 17
- 238000012795 verification Methods 0.000 claims description 12
- 235000014510 cooky Nutrition 0.000 claims description 8
- 238000005516 engineering process Methods 0.000 claims description 6
- 230000008569 process Effects 0.000 claims description 5
- 238000007639 printing Methods 0.000 claims description 3
- 238000003672 processing method Methods 0.000 claims description 2
- 230000007547 defect Effects 0.000 abstract description 2
- 230000009193 crawling Effects 0.000 description 6
- 239000008186 active pharmaceutical agent Substances 0.000 description 4
- 239000003795 chemical substances by application Substances 0.000 description 3
- 238000001514 detection method Methods 0.000 description 2
- 230000006870 function Effects 0.000 description 2
- BUGBHKTXTAQXES-UHFFFAOYSA-N Selenium Chemical compound [Se] BUGBHKTXTAQXES-UHFFFAOYSA-N 0.000 description 1
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 230000010485 coping Effects 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 238000005096 rolling process Methods 0.000 description 1
- 229910052711 selenium Inorganic materials 0.000 description 1
- 239000011669 selenium Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/44—Arrangements for executing specific programs
- G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
- G06F9/45504—Abstract machines for programme code execution, e.g. Java virtual machine [JVM], interpreters, emulators
- G06F9/45529—Embedded in an application, e.g. JavaScript in a Web browser
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention relates to a browser page data acquisition method, terminal equipment and a storage medium, wherein the method comprises the following steps: s1: starting a background program of the browser extension, and polling a task list of a task center to acquire a task; s2: the browser expands a browser page corresponding to the opened task, judges whether the page access is normal, and enters S4 when the page access is normal; when the page cannot be accessed or the access is wrong, giving out error information, and ending; when the page access is restricted, proceed to S3; s3: according to the type of the limited page access, after the limited page access state is removed by adopting a corresponding rule, the page is normally accessed, and the step S4 is entered; s4: executing a JavaScript script in the page to acquire data required to be acquired in the page; s5: and correspondingly processing the data acquisition result according to a processing mode configured in the browser extension. The invention collects internet data through browser extension, overcomes the defects of various crawler tools in the past and simulates the browser request in hundreds.
Description
Technical Field
The invention relates to the field of data acquisition, in particular to a browser page data acquisition method, terminal equipment and a storage medium.
Background
In the existing web page data crawler technology and scheme, the main technical difficulties are a reverse crawling strategy, such as that access is prohibited when IP (Internet protocol) is frequently grabbed, templates are irregularly changed, and URL grabbing fails. Aiming at the technical difficulties, the method has a plurality of coping strategies, and can use a browser to access in a mode aiming at a reverse crawling strategy; multiple sets of template rules and strategies can be set for the regular change of the template; for the disabled IP detection, a proxy IP library can be purchased, or a plurality of applications are deployed to grab respectively, the access frequency of a single node is reduced, and the page access interval time is set. However, the implementation manner of the above strategy is complex and the effect is not good.
With the development of browser technology and JS (JavaScript, a widely-used high-level programming language of just-in-time compilation), various crawling technologies emerge endlessly, where front-end crawling is the first threshold. General practice of web crawlers: compiling the crawler based on Socket communication; compiling the crawler based on the HttpURLCONnection class; compiling a crawler based on an Apache HttpClient package; headless (interface-less) browsers based on phantomjs and the like; based on a top (interfaced) browser like the Selenium. The method cannot simulate the behavior characteristics of the browser in a hundred percent if the method is detected and blocked by the opposite side.
Disclosure of Invention
In order to solve the above problems, the present invention provides a browser page data acquisition method, a terminal device, and a storage medium.
The specific scheme is as follows:
a browser page data acquisition method comprises the following steps:
s1: starting a background program of the browser extension, and polling a task list of a task center to acquire a page data acquisition task;
s2: the browser extension opens a browser page according to the URL link contained in the task, judges whether the page access is normal, and enters S4 when the page access is normal; when the page cannot be accessed or the access is wrong, giving out error information, and ending; when the page access is restricted, proceed to S3;
s3: according to the type of the limited page access, after the limited page access state is removed by adopting a corresponding rule, the page is normally accessed, and the step S4 is entered;
s4: executing a JavaScript script in the page to acquire data required to be acquired in the page;
s5: and correspondingly processing the data acquisition result according to a processing mode configured in the browser extension.
Further, step S1, after opening the daemon of the browser extension, further includes configuring the browser extension, including:
(1) configuring less than 1000 tab pages which are opened simultaneously on each server;
(2) configuring automatic identification and removing interference information in the page;
(3) configuring and utilizing a browser API to control a browser to open and close a target webpage;
(4) configuring and utilizing a browser API and a JavaScript script to perform page operation;
(5) and configuring a data acquisition result processing mode.
Further, the rule for releasing the page access restricted state in step S3 includes:
when the limited type of the page access is that the IP is limited, reducing the access frequency of the webpage, or constructing an IP agent pool and replacing the IP for access;
when the page access limited type is the type needing the verification code, a mode of bypassing the verification code and automatically identifying or manually inputting the verification code image is adopted;
when the limited type of the page access is the type of the Cookie needing to be logged in, an account number needed by the page is registered in advance, configuration is carried out in the browser extension, and when the Cookie needing to be logged in, login is automatically simulated or the account number is manually input for login according to the configuration.
Further, step S4 includes saving the captured text, the table data, the picture link, and the video link as the text data in the JSON format, and saving the captured picture and video as the original format of the picture and video.
Further, the process of acquiring the data to be acquired in the page in step S4 includes: automatically identifying the paging mode of the page, and automatically clicking and opening the next page according to the identified paging mode; by simulating mouse scrolling to obtain more data per page.
Further, in step S4, when the collected data includes a link, the link is extracted to generate a new task, and the new task is sent to the task center, and the task center performs rearrangement on the received task and then adds the task to the task list.
Further, the processing method of the data acquisition result in step S5 includes: and sending to a server, saving, exporting and printing.
Further, the method also comprises the step of S6: after the task processing technology, the browser page is closed, and the resource occupation is destroyed.
A browser page data acquisition terminal device comprises a processor, a memory and a computer program which is stored in the memory and can run on the processor, wherein the steps of the method of the embodiment of the invention are realized when the processor executes the computer program.
A computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of the method as described above for an embodiment of the invention.
According to the technical scheme, the Internet data are collected through browser extension, the defects of various crawler tools are overcome in the past, and the browser request is simulated in hundreds.
Drawings
Fig. 1 is a flowchart illustrating a first embodiment of the present invention.
Detailed Description
To further illustrate the various embodiments, the invention provides the accompanying drawings. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the embodiments. Those skilled in the art will appreciate still other possible embodiments and advantages of the present invention with reference to these figures.
The invention will now be further described with reference to the accompanying drawings and detailed description.
The first embodiment is as follows:
the embodiment of the invention provides a browser page data acquisition method, as shown in fig. 1, the method comprises the following steps:
s1: and starting a background program of the browser extension, and polling a background server to acquire a page data acquisition task.
The data format of the task is as follows:
the browser extension has a plurality of configuration options, and after a background program of the browser extension is opened, the browser extension is configured, wherein the configuration options include:
(1) less than 1000 tab pages are configured to be simultaneously open on each server.
The browser extension can simultaneously open and close a plurality of tab pages, and in order to ensure that the system stably runs on a single server, the embodiment limits the number of tab pages to be opened simultaneously to be less than 1000.
(2) The configuration automatically identifies and removes the interference information in the page.
The interference information comprises advertisement, promotion and the like, and abnormal data can be prevented from being collected by removing the interference information.
(3) The configuration controls the browser to open and close the target web page using the browser API.
(4) And configuring and utilizing a browser API and a JavaScript script to perform page operation.
The page operations include clicking, scrolling, etc.
(5) And configuring a data acquisition result processing mode.
The processing mode of the data acquisition result comprises sending to a server, storing, exporting, printing and the like.
S2: the browser extension opens a browser page according to the URL link contained in the task, judges whether the page access is normal, and enters S4 when the page access is normal; when the page cannot be accessed or the access is wrong, giving out error information, and ending; when the page access is restricted, S3 is entered.
Types of restricted page access include IP restricted, requiring a passcode, requiring a login Cookie, etc.
S3: and according to the type of the limited page access, after the limited page access state is removed by adopting the corresponding rule, the page is normally accessed, and the step S4 is entered.
The rule for releasing the page access restricted state in the embodiment comprises:
(1) and when the limited page access type is IP limited, reducing the access frequency of the webpage, or constructing an IP agent pool and replacing the IP for access.
(2) When the page access limited type is that the verification code is needed, the verification code is bypassed, and the verification code image is automatically identified or manually input.
(3) When the limited type of the page access is the type of the Cookie needing to be logged in, an account number needed by the page is registered in advance, configuration is carried out in the browser extension, and when the Cookie needing to be logged in, login is automatically simulated or the account number is manually input for login according to the configuration.
S4: and executing the JavaScript script in the page to collect the data needing to be collected in the page.
The JavaScript script has two sources, one is user-defined and is stored in a match-format field in the task; one is smart identification such as list data identification < ul > < li > tag, link identification < a href >, title, content/attribute, etc.
The storage format of the collected data is as follows: and storing the collected text, table data, picture link and video link as text data in a JSON format, and storing the collected picture and video as original formats of the picture and video.
The process of the data needing to be collected in the page comprises the following steps: automatically identifying the paging mode of the page, and automatically clicking and opening the next page according to the identified paging mode; by simulating mouse scrolling to obtain more data per page. By opening the next page and simulating the mouse to roll, the data in one webpage can be completely identified, and the omission of the data is avoided. To avoid large data loads, the page number of pages may be set.
When the collected data comprises the link, the link is extracted to generate a new task, the new task is sent to a task center, and the task center performs repetition elimination on the received task and then adds the task into a task list.
And identifying the connection in the data in a mode of constructing a regular expression.
Further, the method comprises a plurality of task lists, each task is randomly added into a different task list, and when the task is acquired, each task list is polled to acquire the task. And a task re-ranking rule is set to avoid repeatedly acquiring tasks. And when the task is added into the task list, setting the priority of each task, and acquiring the task according to the priority.
S5: and correspondingly processing the data acquisition result according to a processing mode configured in the browser extension.
Furthermore, the method also comprises the step of recording the processed tasks to avoid repeated tasks.
S6: after the task processing technology, the browser page is closed, and the resource occupation is destroyed.
The embodiment of the invention automatically opens and closes the page through the browser extension and directly processes the loaded dynamic page content. Automatically identifying multi-level links and performing deep crawling; supporting distributed and multi-task acquisition by adopting a queue mode; the page can automatically extract key target data and support user-defined acquisition requirements; the automatic loading dynamic IP agent and the manual verification of the verification code are supported, and the problems of website reverse crawling and access limitation are solved. By the method, the operations of user login, page clicking, refreshing, paging, rolling and the like can be completely simulated, and the target data can be rapidly and intelligently extracted. The method breaks through the safety limit of the website, and is easier to process the problems of IP detection, Cookie authentication, verification codes and the like. In other embodiments, a third-party plug-in may also be introduced to implement, for example, a page screenshot, picture content identification, and the like, so as to implement different acquisition requirements.
Example two:
the invention also provides browser page data acquisition terminal equipment which comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, wherein the steps in the method embodiment of the first embodiment of the invention are realized when the processor executes the computer program.
Further, as an executable scheme, the browser page data acquisition terminal device may be a desktop computer, a notebook, a palm computer, a cloud server, or other computing devices. The browser page data acquisition terminal device can include, but is not limited to, a processor and a memory. It is understood by those skilled in the art that the above-mentioned constituent structure of the browser page data acquisition terminal device is only an example of the browser page data acquisition terminal device, and does not constitute a limitation on the browser page data acquisition terminal device, and may include more or less components than the above, or combine some components, or different components, for example, the browser page data acquisition terminal device may further include an input/output device, a network access device, a bus, and the like, which is not limited in this embodiment of the present invention.
Further, as an executable solution, the Processor may be a Central Processing Unit (CPU), other general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, a discrete Gate or transistor logic device, a discrete hardware component, and the like. The general processor can be a microprocessor or the processor can be any conventional processor, and the processor is a control center of the browser page data acquisition terminal device and is connected with various parts of the whole browser page data acquisition terminal device by various interfaces and lines.
The memory can be used for storing the computer program and/or the module, and the processor realizes various functions of the browser page data acquisition terminal equipment by running or executing the computer program and/or the module stored in the memory and calling data stored in the memory. The memory can mainly comprise a program storage area and a data storage area, wherein the program storage area can store an operating system and an application program required by at least one function; the storage data area may store data created according to the use of the mobile phone, and the like. In addition, the memory may include high speed random access memory, and may also include non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), at least one magnetic disk storage device, a Flash memory device, or other volatile solid state storage device.
The invention also provides a computer-readable storage medium, in which a computer program is stored, which, when being executed by a processor, carries out the steps of the above-mentioned method of an embodiment of the invention.
The module/unit integrated with the browser page data acquisition terminal device can be stored in a computer readable storage medium if the module/unit is implemented in the form of a software functional unit and sold or used as an independent product. Based on such understanding, all or part of the flow of the method according to the embodiments of the present invention may also be implemented by a computer program, which may be stored in a computer-readable storage medium, and when the computer program is executed by a processor, the steps of the method embodiments may be implemented. Wherein the computer program comprises computer program code, which may be in the form of source code, object code, an executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), software distribution medium, and the like.
While the invention has been particularly shown and described with reference to a preferred embodiment, it will be understood by those skilled in the art that various changes in form and detail may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
Claims (10)
1. A browser page data acquisition method is characterized by comprising the following steps:
s1: starting a background program of the browser extension, and polling a task list of a task center to acquire a page data acquisition task;
s2: the browser extension opens a browser page according to the URL link contained in the task, judges whether the page access is normal, and enters S4 when the page access is normal; when the page cannot be accessed or the access is wrong, giving out error information, and ending; when the page access is restricted, proceed to S3;
s3: according to the type of the limited page access, after the limited page access state is removed by adopting a corresponding rule, the page is normally accessed, and the step S4 is entered;
s4: executing a JavaScript script in the page to acquire data required to be acquired in the page;
s5: and correspondingly processing the data acquisition result according to a processing mode configured in the browser extension.
2. The browser page data acquisition method of claim 1, wherein: step S1, after opening the daemon of the browser extension, further includes configuring the browser extension, including:
(1) configuring less than 1000 tab pages which are opened simultaneously on each server;
(2) configuring automatic identification and removing interference information in the page;
(3) configuring and utilizing a browser API to control a browser to open and close a target webpage;
(4) configuring and utilizing a browser API and a JavaScript script to perform page operation;
(5) and configuring a data acquisition result processing mode.
3. The browser page data acquisition method of claim 1, wherein: the rule for releasing the page access restricted state in step S3 includes:
when the limited type of the page access is that the IP is limited, reducing the access frequency of the webpage, or constructing an IP agent pool and replacing the IP for access;
when the page access limited type is the type needing the verification code, a mode of bypassing the verification code and automatically identifying or manually inputting the verification code image is adopted;
when the limited type of the page access is the type of the Cookie needing to be logged in, an account number needed by the page is registered in advance, configuration is carried out in the browser extension, and when the Cookie needing to be logged in, login is automatically simulated or the account number is manually input for login according to the configuration.
4. The browser page data acquisition method of claim 1, wherein: step S4 further includes saving the collected text, table data, picture link, and video link as text data in JSON format, and saving the collected picture and video as original formats of picture and video.
5. The browser page data acquisition method of claim 1, wherein: the process of acquiring the data to be acquired in the page in step S4 includes: automatically identifying the paging mode of the page, and automatically clicking and opening the next page according to the identified paging mode; by simulating mouse scrolling to obtain more data per page.
6. The browser page data acquisition method of claim 1, wherein: in step S4, when the acquired data includes a link, the link is extracted to generate a new task, and the new task is sent to the task center, and the task center performs deduplication on the received task and adds the duplicate to the task list.
7. The browser page data acquisition method of claim 1, wherein: the processing method of the data acquisition result in step S5 includes: and sending to a server, saving, exporting and printing.
8. The browser page data acquisition method of claim 1, wherein: further comprising S6: after the task processing technology, the browser page is closed, and the resource occupation is destroyed.
9. The utility model provides a browser page data acquisition terminal equipment which characterized in that: comprising a processor, a memory and a computer program stored in the memory and running on the processor, the processor implementing the steps of the method according to any one of claims 1 to 8 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: the computer program when executed by a processor implementing the steps of the method as claimed in any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110158353.4A CN112800311A (en) | 2021-02-05 | 2021-02-05 | Browser page data acquisition method, terminal device and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110158353.4A CN112800311A (en) | 2021-02-05 | 2021-02-05 | Browser page data acquisition method, terminal device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112800311A true CN112800311A (en) | 2021-05-14 |
Family
ID=75814316
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110158353.4A Pending CN112800311A (en) | 2021-02-05 | 2021-02-05 | Browser page data acquisition method, terminal device and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112800311A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292647A (en) * | 2022-10-08 | 2022-11-04 | 北京易特思维信息技术有限公司 | Non-invasive government data acquisition method |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012146100A (en) * | 2011-01-11 | 2012-08-02 | Fujitsu Ltd | Attack imitation testing method, attack imitation testing device and attack imitation testing program |
| CN105512193A (en) * | 2015-11-26 | 2016-04-20 | 上海携程商务有限公司 | Data acquisition system and method based on browser expansion |
| CN105956175A (en) * | 2016-05-24 | 2016-09-21 | 考拉征信服务有限公司 | Webpage content crawling method and device |
| CN108932285A (en) * | 2018-05-22 | 2018-12-04 | 北京工业大学 | A kind of data grab method and system based on browser extension |
| CN110276041A (en) * | 2019-07-01 | 2019-09-24 | 浪潮卓数大数据产业发展有限公司 | A kind of web data acquisition methods and system based on Google's browser plug-in |
| CN110909229A (en) * | 2019-11-27 | 2020-03-24 | 佛山科学技术学院 | Webpage data acquisition and storage system based on simulated browser access |
-
2021
- 2021-02-05 CN CN202110158353.4A patent/CN112800311A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2012146100A (en) * | 2011-01-11 | 2012-08-02 | Fujitsu Ltd | Attack imitation testing method, attack imitation testing device and attack imitation testing program |
| CN105512193A (en) * | 2015-11-26 | 2016-04-20 | 上海携程商务有限公司 | Data acquisition system and method based on browser expansion |
| CN105956175A (en) * | 2016-05-24 | 2016-09-21 | 考拉征信服务有限公司 | Webpage content crawling method and device |
| CN108932285A (en) * | 2018-05-22 | 2018-12-04 | 北京工业大学 | A kind of data grab method and system based on browser extension |
| CN110276041A (en) * | 2019-07-01 | 2019-09-24 | 浪潮卓数大数据产业发展有限公司 | A kind of web data acquisition methods and system based on Google's browser plug-in |
| CN110909229A (en) * | 2019-11-27 | 2020-03-24 | 佛山科学技术学院 | Webpage data acquisition and storage system based on simulated browser access |
Non-Patent Citations (1)
| Title |
|---|
| 吴仲治: "《Python开发技术大全》", 31 August 2020, 北京：机械工业出版社 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115292647A (en) * | 2022-10-08 | 2022-11-04 | 北京易特思维信息技术有限公司 | Non-invasive government data acquisition method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102200971B (en) | Method and equipment for realizing webpage content previewing | |
| CN110826006B (en) | Abnormal collection behavior identification method and device based on privacy data protection | |
| CN110929145B (en) | Public opinion analysis method, public opinion analysis device, computer device and storage medium | |
| CN106682124A (en) | Image recognition method, device and equipment | |
| CN111737692B (en) | Application program risk detection method and device, equipment and storage medium | |
| CN109710224B (en) | Page processing method, device, equipment and storage medium | |
| CN110555179A (en) | Dynamic website script evidence obtaining method, terminal equipment and storage medium | |
| CN113704590A (en) | Webpage data acquisition method and device, electronic equipment and storage medium | |
| CN102880698B (en) | A kind of crawl website defining method and device | |
| CN111355628B (en) | Model training method, service identification method, device and electronic device | |
| CN112800311A (en) | Browser page data acquisition method, terminal device and storage medium | |
| CN110737814A (en) | Crawling method and device for website data, electronic equipment and storage medium | |
| CN112269666B (en) | Applet dead-link detection method and device, computing device and medium | |
| CN111859069B (en) | Network malicious crawler identification method, system, terminal and storage medium | |
| CN111026945B (en) | Multi-platform crawler scheduling method, device and storage medium | |
| CN108062459B (en) | Method and device for preventing page information from being captured | |
| CN117033249A (en) | Test case generation method and device, computer equipment and storage medium | |
| CN109669799B (en) | Error reporting control method, device and storage medium | |
| CN116343221A (en) | Certificate information automatic input method and device, electronic equipment and storage medium | |
| CN115758011A (en) | Data unloading method, data display method, device, equipment and storage medium | |
| CN109145307A (en) | User's face sketch recognition method, method for pushing, device, equipment and storage medium | |
| CN110457632B (en) | Webpage loading processing method and device | |
| CN113569552A (en) | Log template extraction method and device, electronic equipment and computer storage medium | |
| JP6763433B2 (en) | Information gathering system, information gathering method, and program | |
| CN111783006A (en) | Page generation method and device, electronic equipment and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20210514 |