CN107948052A - Information crawler method, apparatus, electronic equipment and system - Google Patents
Information crawler method, apparatus, electronic equipment and system Download PDFInfo
- Publication number
- CN107948052A CN107948052A CN201711125869.9A CN201711125869A CN107948052A CN 107948052 A CN107948052 A CN 107948052A CN 201711125869 A CN201711125869 A CN 201711125869A CN 107948052 A CN107948052 A CN 107948052A
- Authority
- CN
- China
- Prior art keywords
- public platform
- request
- address
- server
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L51/00—User-to-user messaging in packet-switching networks, transmitted according to store-and-forward or real-time protocols, e.g. e-mail
- H04L51/04—Real-time or near real-time messaging, e.g. instant messaging [IM]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/955—Retrieval from the web using information identifiers, e.g. uniform resource locators [URL]
- G06F16/9566—URL specific, e.g. using aliases, detecting broken or misspelled links
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/14—Session management
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The present invention provides a kind of information crawler method, apparatus, electronic equipment and system, belong to Internet communication technology field.Information crawler method, apparatus, electronic equipment and system provided in an embodiment of the present invention, electronic equipment is set between wechat client and wechat server, obtain the network request address that wechat server provides, it is supplied to web crawlers unit, the data and information in wechat public platform are crawled by web crawlers unit, all articles of wechat public platform can be obtained, crawlers is obtained the content of substantial amounts of wechat public platform in a short time, the performance and efficiency of crawlers are lifted in very big program.
Description
Technical field
The present invention relates to Internet communication technology field, in particular to a kind of information crawler method, apparatus, electronics
Equipment and system.
Background technology
The mode of checking of the article in wechat public platform has two kinds at present：A kind of looked into by the function of search of search dog wechat
See public platform article, one kind is to check public platform article by handset Wechat APP.Existing wechat public platform crawlers are most
Number is crawled by way of search dog searches for wechat public platform, and efficiency is low, and search dog search has stringent anti-reptile rule, no
The article of a large amount of public platforms can be obtained in a short time, even if it is also to have quantity limitation to get wechat public platform article.
The content of the invention
For above-mentioned problems of the prior art, the present invention provides a kind of information crawler method, apparatus, electronics to set
Standby and system.
In a first aspect, an embodiment of the present invention provides a kind of information crawler method, applied to be arranged on wechat client with
On electronic equipment between wechat server, for crawling the content in wechat public platform, including：
When receiving the wechat public platform access request that wechat client is sent, the chain that the access request carries is extracted
It is grounded location and wechat public platform mark；
To the chained address, accordingly wechat server sends linking request；The linking request includes the wechat
Public platform identifies；
Receive the wechat public platform that wechat server returns and identify corresponding network request address；
The network request address is sent to the web crawlers unit for crawling information, so that the web crawlers list
Member crawls the content in the wechat public platform.
With reference to first aspect, an embodiment of the present invention provides the first possible embodiment of first aspect, wherein, to
The chained address accordingly wechat server send linking request the step of, including：
Wechat program is opened, accordingly wechat server transmission link please to the chained address by the wechat program
Ask.
With reference to first aspect, an embodiment of the present invention provides second of possible embodiment of first aspect, wherein, institute
The method of stating further includes：
When receiving the wechat public platform access request that wechat client is sent, the use that the access request carries is extracted
Family information；
Wechat program is logged according to the user information, by the wechat program to the chained address accordingly wechat
Server sends linking request.
With reference to first aspect, an embodiment of the present invention provides the third possible embodiment of first aspect, wherein, connect
After receiving the step of wechat public platform that wechat server returns identifies corresponding network request address, the method is also wrapped
Include：
The network request address is sent to the wechat client for initiating wechat public platform access request.
Second aspect, the embodiment of the present invention additionally provide a kind of information crawler device, applied to being arranged on wechat client
On electronic equipment between wechat server, for crawling the content in wechat public platform, including：
Chained address acquiring unit, for when receiving the wechat public platform access request that wechat client is sent, carrying
Take the chained address and wechat public platform mark that the access request carries；
Linking request transmitting element, for accordingly wechat server to send linking request to the chained address；It is described
Linking request includes the wechat public platform mark；
Request address transmitting element, the wechat public platform for receiving wechat server return identify corresponding network
Request address；The network request address is sent to the web crawlers unit for crawling information, so that the web crawlers
Unit crawls the content in the wechat public platform.
With reference to second aspect, an embodiment of the present invention provides the first possible embodiment of second aspect, wherein, institute
Linking request transmitting element is stated, is additionally operable to：
Wechat program is opened, accordingly wechat server transmission link please to the chained address by the wechat program
Ask.
With reference to second aspect, an embodiment of the present invention provides second of possible embodiment of second aspect, wherein, institute
Request address transmitting element is stated, is additionally operable to：
The network request address is sent to the wechat client for initiating wechat public platform access request.
The third aspect, the embodiment of the present invention additionally provide a kind of electronic equipment, including processor and memory；The storage
Device is used to store the program for supporting processor to perform the above method；The processor is configurable for performing in the memory
The program of storage.
Fourth aspect, the embodiment of the present invention additionally provide a kind of information crawler system, including one or more wechat clients
End, wechat server and above-mentioned electronic equipment；The wechat client connects the wechat service by the electronic equipment
Device.
5th aspect, the embodiment of the present invention additionally provide a kind of machinable medium, it is characterised in that are stored with
State the computer software instructions used in device.
The embodiment of the present invention brings following beneficial effect：
Information crawler method, apparatus, electronic equipment and system provided in an embodiment of the present invention, in wechat client and wechat
Electronic equipment is set between server, obtains the network request address that wechat server provides, there is provided web crawlers unit is given, by
Web crawlers unit crawls data and information in wechat public platform, can obtain all articles of wechat public platform, make reptile
Program can obtain the content of substantial amounts of wechat public platform in a short time, in very big program lifted crawlers performance with
Efficiency.
Other features and advantages of the present invention will illustrate in the following description, also, partly become from specification
Obtain it is clear that or being understood by implementing the present invention.The purpose of the present invention and other advantages are in specification, claims
And specifically noted structure is realized and obtained in attached drawing.
To enable the above objects, features and advantages of the present invention to become apparent, preferred embodiment cited below particularly, and coordinate
Appended attached drawing, is described in detail below.
Brief description of the drawings
, below will be to specific in order to illustrate more clearly of the specific embodiment of the invention or technical solution of the prior art
Embodiment or attached drawing needed to be used in the description of the prior art are briefly described, it should be apparent that, in describing below
Attached drawing is some embodiments of the present invention, for those of ordinary skill in the art, before not making the creative labor
Put, other attached drawings can also be obtained according to these attached drawings.
The flow chart for the information crawler method that Fig. 1 is provided by one embodiment of the invention；
The flow chart for the information crawler method that Fig. 2 is provided by another embodiment of the present invention；
The structure diagram for the information crawler device that Fig. 3 is provided by one embodiment of the invention；
The structure diagram for the information crawler system that Fig. 4 is provided by one embodiment of the invention.
Embodiment
To make the purpose, technical scheme and advantage of the embodiment of the present invention clearer, below in conjunction with attached drawing to the present invention
Technical solution be clearly and completely described, it is clear that described embodiment is part of the embodiment of the present invention, rather than
Whole embodiments.The component of embodiments of the present invention, which are generally described and illustrated herein in the accompanying drawings can be matched somebody with somebody with a variety of
Put to arrange and design.Therefore, the detailed description of the embodiment of the present invention to providing in the accompanying drawings is not intended to limit below
The scope of claimed invention, but it is merely representative of the selected embodiment of the present invention.Based on the embodiments of the present invention, originally
Field those of ordinary skill all other embodiments obtained without making creative work, belong to the present invention
The scope of protection.
For present crawlers crawl wechat public platform article efficiency it is low and can crawl article limited amount the problem of,
An embodiment of the present invention provides a kind of information crawler method, apparatus, electronic equipment and system, below first to the information of the present invention
Crawling method describes in detail.
Embodiment one
This embodiment offers a kind of information crawler method, applied to being arranged between wechat client and wechat server
Electronic equipment on, for crawling the content in wechat public platform.Fig. 1 shows the information crawler method that the present embodiment is provided
Flow chart.As shown in Figure 1, this method comprises the following steps：
Step S101, when receiving the wechat public platform access request that wechat client is sent, extraction access request is taken
The chained address of band and wechat public platform mark.
In the present embodiment, wechat client needs above-mentioned electronic equipment to be carried out as intermediate equipment with wechat server
Connection.The electronic equipment is referred to as proxy server.When wechat client pays close attention to certain wechat public platform or checks certain wechat
, it is necessary to send wechat public platform access request during certain article in public platform, after which is received by electronic equipment,
Electronic equipment can forward the access request.
Step S102, to chained address, accordingly wechat server sends linking request；Linking request includes wechat public affairs
Many numbers marks.
When sending linking request to wechat server, it can be transmitted by third party's program or wechat program.For example,
Wechat program is opened, accordingly wechat server sends linking request to chained address by wechat program.
Alternatively, when receiving the wechat public platform access request that wechat client is sent, extraction access request carries
User information；Wechat program is logged according to user information, accordingly wechat server is sent out to chained address by wechat program
Send linking request.By this operation, wechat server can determine which the request comes from from the linking request received
Wechat client.
Step S103, receives the wechat public platform that wechat server returns and identifies corresponding network request address.
, can be according to the wechat public platform identifier lookup wechat public platform or this is micro- after wechat server receives linking request
Believe the network request address of public platform article, which is returned back into electronic equipment.
Step S104, network request address is sent to the web crawlers unit for crawling information, so that web crawlers
Unit crawls the content in wechat public platform.
Information crawler method provided in this embodiment, electronic equipment is set between wechat client and wechat server,
Obtain the network request address that wechat server provides, there is provided give web crawlers unit, wechat public affairs are crawled by web crawlers unit
Data and information in many numbers, can obtain all articles of wechat public platform, crawlers is obtained in a short time
The content of substantial amounts of wechat public platform, lifts the performance and efficiency of crawlers in very big program.
In one more preferably embodiment, as shown in Fig. 2, information crawler method includes the following steps：
Step S201, when receiving the wechat public platform access request that wechat client is sent, extraction access request is taken
The chained address of band and wechat public platform mark；
Step S202, to chained address, accordingly wechat server sends linking request；Linking request includes wechat public affairs
Many numbers marks；
Step S203, receives the wechat public platform that wechat server returns and identifies corresponding network request address；
Step S204, network request address is sent to the web crawlers unit for crawling information, so that web crawlers
Unit crawls the content in wechat public platform；
Step S205, network request address is sent to the wechat client for initiating wechat public platform access request.
Wherein, the order of step S204 and step S205 can exchange.
A kind of concrete application of information crawler method provided in an embodiment of the present invention is implemented as follows：
Electronic equipment passes through MITM (Man-in-the-middle-attacks, man-in-the-middle attack) technical limit spacing wechat visitor
The network request address that the wechat public platform access request and wechat server at family end return.MITM technologies are a kind of indirect aggressions
The network technology of computer, although MITM technologies bring definitely Network Security Vulnerabilities, reasonably utilizes MITM technologies, can
Greatly to simplify the development process of reptile, the performance and efficiency of crawler technology are lifted.
When the information crawler method of the present embodiment is realized using MITM technologies, first, wechat client (user terminal) needs
Open MITM services.Then the proxy server (electronic equipment) of wechat client is set, take the MITM of wechat client
Business accesses wechat server by this proxy server.After being provided with, open wechat program in wechat client and log in,
Wechat client, which accesses, needs the wechat public platform that reptile crawls.MITM services on proxy server can obtain wechat public affairs
The network request address of many numbers, then sends the network request address to web crawlers unit, so that web crawlers unit can
To crawl the content of wechat public platform.
Information crawler method provided in this embodiment, electronic equipment is set between wechat client and wechat server,
Obtain the network request address that wechat server provides, there is provided give web crawlers unit, wechat public affairs are crawled by web crawlers unit
Data and information in many numbers, can obtain all articles of wechat public platform, crawlers is obtained in a short time
The content of substantial amounts of wechat public platform, lifts the performance and efficiency of crawlers in very big program.
Embodiment two
With the method embodiment accordingly, a kind of information crawler device is present embodiments provided, applied to being arranged on
On electronic equipment between wechat client and wechat server, for crawling the content in wechat public platform.Fig. 3 shows this
The structure diagram for the information crawler device that embodiment is provided.As shown in figure 3, the device includes：
Chained address acquiring unit 31, for when receiving the wechat public platform access request that wechat client is sent,
Extract chained address and the wechat public platform mark that the access request carries；
Linking request transmitting element 32, for accordingly wechat server to send linking request to the chained address；Institute
State linking request and include the wechat public platform mark；
Request address transmitting element 33, the wechat public platform for receiving wechat server return identify corresponding net
Network request address；The network request address is sent to the web crawlers unit for crawling information, so that the network is climbed
Worm unit crawls the content in the wechat public platform.
Wherein, chained address acquiring unit 31, can be also used for the user information that extraction access request carries, the user
Information can include one or more of：User name, password, the pet name, telephone number, mailbox etc..
Linking request transmitting element 32, can be also used for opening wechat program, by the wechat program to the link
Accordingly wechat server sends linking request for address；Or for logging in wechat program according to user information, pass through wechat journey
To chained address, accordingly wechat server sends linking request to sequence.
Request address transmitting element 33, can be also used for sending to initiation wechat public platform to access by network request address asking
The wechat client asked.
Information crawler device provided in an embodiment of the present invention, sets electronics to set between wechat client and wechat server
It is standby, obtain the network request address that wechat server provides, there is provided give web crawlers unit, wechat is crawled by web crawlers unit
Data and information in public platform, can obtain all articles of wechat public platform, crawlers is obtained in a short time
The content of substantial amounts of wechat public platform is taken, the performance and efficiency of crawlers are lifted in very big program.
Embodiment three
This embodiment offers a kind of electronic equipment.The electronic equipment includes processor and memory.
Memory is used for the software program module for storing processor execution.Processor is used to perform institute in above-described embodiment one
The method of record.Memory may include high speed random access memory, may also include nonvolatile memory, such as one or more magnetic
Property storage device, flash memory or other non-volatile solid state memories.
Further, above-mentioned software program module may also include：Operating system and service module.Wherein operating system,
LINUX, UNIX, WINDOWS are may be, for example, it may include various to be used for management system task (such as memory management, storage device
Control, power management etc.) component software and/or driving, and can mutually be communicated with various hardware or component software, so as to provide
The running environment of other software component.
Service module is operated on the basis of operating system, and is monitored by the network service of operating system come automatic network
Request, corresponding data processing is completed according to request, and returns to mobile terminal of the handling result to user.That is, service
Module is used to provide network service to the mobile terminal of user.
The electronic equipment further includes the mixed-media network modules mixed-media for receiving and sending network signal.Above-mentioned network signal may include
Wireless signal or wire signal.
It is understood that the explanation is only to schematically illustrate, server may also include than above-mentioned more or less
Component, or there is different configurations.Above-mentioned each component can use hardware, software or its combination to realize.It is in addition, of the invention
Server in embodiment can also include the server of multiple specific difference in functionality.
The technique effect and preceding method embodiment of the electronic equipment that above-described embodiment is provided, its realization principle and generation
Identical, to briefly describe, device embodiment part does not refer to part, refers to corresponding contents in preceding method embodiment.
Example IV
This embodiment offers a kind of information crawler system, Fig. 4 shows the structure diagram of the information crawler system.Such as figure
Shown in 4, which includes one or more wechat client 100, wechat server 300 and above-mentioned electronic equipments
200；Wechat client 100 connects wechat server 300 by electronic equipment 200.
When wechat client sends wechat public platform access request to wechat server, electronic equipment can be according to the visit
Ask that request sends linking request to wechat server, obtain the network request address for the wechat public platform that wechat server returns,
The network request address is supplied to web crawlers unit, to crawl the content of wechat public platform.
Further, another embodiment of the present invention additionally provides a kind of machinable medium, is stored with above device
Computer software instructions used.
Each embodiment in this specification is described by the way of progressive, what each embodiment stressed be with
The difference of other embodiment, between each embodiment identical similar part mutually referring to.
Information crawler method, apparatus, electronic equipment and system provided in an embodiment of the present invention have identical technical characteristic,
So can also solve identical technical problem, reach identical technique effect.
It should be noted that in embodiment provided by the present invention, it should be understood that disclosed system and method, can
To realize by another way.Device embodiment described above is only schematical, for example, the unit is drawn
Point, it is only a kind of division of logic function, there can be other dividing mode when actually realizing, in another example, multiple units or group
Part can combine or be desirably integrated into another system, or some features can be ignored, or not perform.It is described to be used as separation unit
The unit that part illustrates may or may not be physically separate, can be as the component that unit is shown or also may be used
Not to be physical location, you can with positioned at a place, or can also be distributed in multiple network unit.Can be according to reality
Need select some or all of unit therein to realize the purpose of this embodiment scheme.
In addition, each functional unit in embodiment provided by the invention can be integrated in a processing unit, also may be used
To be that unit is individually physically present, can also two or more units integrate in a unit.
If the function is realized in the form of SFU software functional unit and is used as independent production marketing or in use, can be with
It is stored in a computer read/write memory medium.Based on such understanding, technical scheme is substantially in other words
The part to contribute to the prior art or the part of the technical solution can be embodied in the form of software product, the meter
Calculation machine software product is stored in a storage medium, including some instructions are used so that a computer equipment (can be
People's computer, server, or network equipment etc.) perform all or part of step of each embodiment the method for the present invention.
And foregoing storage medium includes：USB flash disk, mobile hard disk, read-only storage (ROM, Read-Only Memory), arbitrary access are deposited
Reservoir (RAM, Random Access Memory), magnetic disc or CD etc. are various can be with the medium of store program codes.
In addition, term " first ", " second ", " the 3rd " are only used for description purpose, and it is not intended that instruction or implying phase
To importance.
Finally it should be noted that：Embodiment described above, is only the embodiment of the present invention, to illustrate the present invention
Technical solution, rather than its limitations, protection scope of the present invention is not limited thereto, although with reference to the foregoing embodiments to this hair
It is bright to be described in detail, it will be understood by those of ordinary skill in the art that：Any one skilled in the art
The invention discloses technical scope in, it can still modify the technical solution described in previous embodiment or can be light
It is readily conceivable that change, or equivalent substitution is carried out to which part technical characteristic；And these modifications, change or replacement, do not make
The essence of appropriate technical solution departs from the spirit and scope of technical solution of the embodiment of the present invention, should all cover the protection in the present invention
Within the scope of.Therefore, protection scope of the present invention answers the scope of the claims of being subject to.
Claims (10)
- A kind of 1. information crawler method, it is characterised in that applied to the electricity being arranged between wechat client and wechat server In sub- equipment, for crawling the content in wechat public platform, including：When receiving the wechat public platform access request that wechat client is sent, the chain ground connection that the access request carries is extracted Location and wechat public platform mark；To the chained address, accordingly wechat server sends linking request；The linking request includes the wechat public Number mark；Receive the wechat public platform that wechat server returns and identify corresponding network request address；The network request address is sent to the web crawlers unit for crawling information, so that the web crawlers unit is climbed Take the content in the wechat public platform.
- 2. according to the method described in claim 1, it is characterized in that, to the chained address accordingly wechat server send chain The step of connecing request, including：Wechat program is opened, accordingly wechat server sends linking request to the chained address by the wechat program.
- 3. according to the method described in claim 1, it is characterized in that, the method further includes：When receiving the wechat public platform access request that wechat client is sent, user's letter that the access request carries is extracted Breath；Wechat program is logged according to the user information, by the wechat program to the chained address accordingly wechat service Device sends linking request.
- 4. according to the method described in claim 1, it is characterized in that, receive the wechat public platform mark that wechat server returns After the step of knowing corresponding network request address, the method further includes：The network request address is sent to the wechat client for initiating wechat public platform access request.
- 5. a kind of information crawler device, it is characterised in that applied to the electricity being arranged between wechat client and wechat server In sub- equipment, for crawling the content in wechat public platform, including：Chained address acquiring unit, for when receiving the wechat public platform access request that wechat client is sent, extracting institute State chained address and the wechat public platform mark of access request carrying；Linking request transmitting element, for accordingly wechat server to send linking request to the chained address；The link Request bag contains the wechat public platform mark；Request address transmitting element, the wechat public platform for receiving wechat server return identify corresponding network request Address；The network request address is sent to the web crawlers unit for crawling information, so that the web crawlers unit Crawl the content in the wechat public platform.
- 6. device according to claim 5, it is characterised in that the linking request transmitting element, is additionally operable to：Wechat program is opened, accordingly wechat server sends linking request to the chained address by the wechat program.
- 7. device according to claim 5, it is characterised in that the request address transmitting element, is additionally operable to：The network request address is sent to the wechat client for initiating wechat public platform access request.
- 8. a kind of electronic equipment, it is characterised in that including processor and memory；The memory, which is used to store, supports processor The program of any one of perform claim requirement 1 to 4 the method；The processor is configurable for performing and is deposited in the memory The program of storage.
- 9. a kind of information crawler system, it is characterised in that will including one or more wechat client, wechat server and rights Seek the electronic equipment described in 8；The wechat client connects the wechat server by the electronic equipment.
- 10. a kind of machinable medium, it is characterised in that be stored with used in any one of claim 5 to 7 described device Computer software instructions.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711125869.9A CN107948052A (en) | 2017-11-14 | 2017-11-14 | Information crawler method, apparatus, electronic equipment and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201711125869.9A CN107948052A (en) | 2017-11-14 | 2017-11-14 | Information crawler method, apparatus, electronic equipment and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107948052A true CN107948052A (en) | 2018-04-20 |
Family
ID=61932177
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201711125869.9A Pending CN107948052A (en) | 2017-11-14 | 2017-11-14 | Information crawler method, apparatus, electronic equipment and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107948052A (en) |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109388735A (en) * | 2018-09-13 | 2019-02-26 | 广州丰石科技有限公司 | A method of crawling wechat public platform information |
| CN110188257A (en) * | 2019-04-16 | 2019-08-30 | 国家计算机网络与信息安全管理中心 | A kind of mobile application collecting method and device |
| CN110677423A (en) * | 2019-09-30 | 2020-01-10 | 深圳前海环融联易信息科技服务有限公司 | Data acquisition method and device based on client agent side and computer equipment |
| CN110781367A (en) * | 2019-09-25 | 2020-02-11 | 中国科学院计算技术研究所 | Internet data acquisition method and system based on man-in-the-middle |
| CN115242491A (en) * | 2022-07-19 | 2022-10-25 | 厦门市美亚柏科信息股份有限公司 | APP cloud detection method and system based on web crawler |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103961869A (en) * | 2014-04-14 | 2014-08-06 | 林云帆 | Device control method |
| CN104035997A (en) * | 2014-06-13 | 2014-09-10 | 淮阴工学院 | Scientific and technical information acquisition and pushing method based on text classification and image deep mining |
| CN105243122A (en) * | 2015-09-29 | 2016-01-13 | 浪潮电子信息产业股份有限公司 | Social software based data acquisition method and apparatus |
| CN105320740A (en) * | 2015-09-22 | 2016-02-10 | 清华大学 | WeChat article and official account acquisition method and acquisition system |
| CN105429865A (en) * | 2015-12-31 | 2016-03-23 | 深圳中泓在线股份有限公司 | WeChat public number data collection method and device based on browser |
| CN105577528A (en) * | 2015-12-31 | 2016-05-11 | 深圳中泓在线股份有限公司 | Wechat official account data collection method and device based on virtual machine |
| CN105718587A (en) * | 2016-01-26 | 2016-06-29 | 王薇 | Network content resource evaluation method and evaluation system |
| CN105790944A (en) * | 2014-12-22 | 2016-07-20 | 深圳易思智科技有限公司 | Wechat-based network authentication method and device |
| CN106202232A (en) * | 2016-06-27 | 2016-12-07 | 中国南方电网有限责任公司电网技术研究中心 | A kind of analysis method and device of power-off event |
-
2017
- 2017-11-14 CN CN201711125869.9A patent/CN107948052A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103961869A (en) * | 2014-04-14 | 2014-08-06 | 林云帆 | Device control method |
| CN104035997A (en) * | 2014-06-13 | 2014-09-10 | 淮阴工学院 | Scientific and technical information acquisition and pushing method based on text classification and image deep mining |
| CN105790944A (en) * | 2014-12-22 | 2016-07-20 | 深圳易思智科技有限公司 | Wechat-based network authentication method and device |
| CN105320740A (en) * | 2015-09-22 | 2016-02-10 | 清华大学 | WeChat article and official account acquisition method and acquisition system |
| CN105243122A (en) * | 2015-09-29 | 2016-01-13 | 浪潮电子信息产业股份有限公司 | Social software based data acquisition method and apparatus |
| CN105429865A (en) * | 2015-12-31 | 2016-03-23 | 深圳中泓在线股份有限公司 | WeChat public number data collection method and device based on browser |
| CN105577528A (en) * | 2015-12-31 | 2016-05-11 | 深圳中泓在线股份有限公司 | Wechat official account data collection method and device based on virtual machine |
| CN105718587A (en) * | 2016-01-26 | 2016-06-29 | 王薇 | Network content resource evaluation method and evaluation system |
| CN106202232A (en) * | 2016-06-27 | 2016-12-07 | 中国南方电网有限责任公司电网技术研究中心 | A kind of analysis method and device of power-off event |
Non-Patent Citations (2)
| Title |
|---|
| LIJINMA: "如何优雅的抓取微信公众号历史文章", 《LARAVEL》 * |
| 飯口組組长: "持续更新，微信公众号文章批量采集系统的构建", 《知乎专栏》 * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109388735A (en) * | 2018-09-13 | 2019-02-26 | 广州丰石科技有限公司 | A method of crawling wechat public platform information |
| CN110188257A (en) * | 2019-04-16 | 2019-08-30 | 国家计算机网络与信息安全管理中心 | A kind of mobile application collecting method and device |
| CN110188257B (en) * | 2019-04-16 | 2021-12-31 | 国家计算机网络与信息安全管理中心 | Mobile application data acquisition method and device |
| CN110781367A (en) * | 2019-09-25 | 2020-02-11 | 中国科学院计算技术研究所 | Internet data acquisition method and system based on man-in-the-middle |
| CN110781367B (en) * | 2019-09-25 | 2023-10-20 | 中国科学院计算技术研究所 | Internet data acquisition method and system based on middleman |
| CN110677423A (en) * | 2019-09-30 | 2020-01-10 | 深圳前海环融联易信息科技服务有限公司 | Data acquisition method and device based on client agent side and computer equipment |
| CN115242491A (en) * | 2022-07-19 | 2022-10-25 | 厦门市美亚柏科信息股份有限公司 | APP cloud detection method and system based on web crawler |
| CN115242491B (en) * | 2022-07-19 | 2024-04-19 | 厦门市美亚柏科信息股份有限公司 | APP cloud detection method and system based on web crawlers |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107948052A (en) | Information crawler method, apparatus, electronic equipment and system | |
| CN106528432B (en) | The construction method and device of test scene data bury a test method | |
| CN104333599B (en) | Share the method and system and application service platform of application | |
| CN104283843B (en) | A kind of method, apparatus and system that user logs in | |
| CN103607385B (en) | Method and apparatus for security detection based on browser | |
| CN107809383A (en) | A kind of map paths method and device based on MVC | |
| CN103150513B (en) | The method of the implantation information in interception application program and device | |
| CN106933871A (en) | Short linking processing method, device and short linked server | |
| CN107404481B (en) | User information recognition methods and device | |
| CN106453216A (en) | Malicious website interception method, malicious website interception device and client | |
| CN108090091A (en) | Web page crawl method and apparatus | |
| CN107147748A (en) | File uploading method and device | |
| CN106878368A (en) | The implementation method and device of information pushing | |
| CN107807937A (en) | A kind of website SEO processing methods, apparatus and system | |
| CN105553968A (en) | Method and device for realizing login by multiple accounts | |
| CN107689941A (en) | A kind of apparatus and method for preventing same user's repeat logon | |
| CN106681799A (en) | Disk inserting method, device and system | |
| CN106572095A (en) | Account registration method, device and system | |
| CN103703474A (en) | Handling device generated data | |
| CN104618410A (en) | Resource push method and resource push device | |
| CN101645021B (en) | Integrating method for multisystem single-spot logging under Java application server | |
| CN106603556A (en) | Single sign-on method, device and system | |
| CN107294905A (en) | A kind of method and device for recognizing user | |
| CN104462488A (en) | Database high reliability solution method and device | |
| CN105144073A (en) | Removable storage device identity and configuration information |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20180420 |