CN108874925A - A kind of distributed vertical crawler method and terminal device - Google Patents
A kind of distributed vertical crawler method and terminal device Download PDFInfo
- Publication number
- CN108874925A CN108874925A CN201810547735.4A CN201810547735A CN108874925A CN 108874925 A CN108874925 A CN 108874925A CN 201810547735 A CN201810547735 A CN 201810547735A CN 108874925 A CN108874925 A CN 108874925A
- Authority
- CN
- China
- Prior art keywords
- platform
- data
- crawl
- task
- grabber
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Abstract
The present invention is suitable for technical field of information retrieval, provides a kind of distributed vertical crawler method and terminal device, including：Crawl task is sent to task distribution platform by middle control platform；Task distribution platform crawls at least one of ability according to the crawl task type of task, the terminal type of data grabber platform, the network type of data grabber platform and data grabber platform, determine the distribution policy of crawl task, and according to distribution policy, crawl task is distributed to data grabber platform；Data grabber platform carries out data grabber according to crawl task, and crawl result is sent to Data Analysis Platform；Data Analysis Platform as a result, the data that pre-set of load extract strategy, judges whether there is new crawl task, if so, new crawl business is distributed to data grabber platform by task distribution platform, if nothing, will grab result and be sent to middle control platform according to crawl.Crawl efficiency lower problem in crawler end when huge that this method solve data volumes.
Description
Technical field
The invention belongs to technical field of information retrieval more particularly to a kind of distributed vertical crawler method and terminal devices.
Background technique
With the rapid development of network, WWW becomes the carrier of bulk information, how to efficiently extract and use these
Information becomes a huge challenge.The tool that search engine assists people to retrieve information as one becomes ten thousand dimension of user's access
The entrance and guide of net.But there is also certain limitations for existing versatility search engine, such as：Different field, difference
The user of background often has different a retrieval purpose and demand, the result that universal search engine is returned include a large number of users not
The webpage of care；The target of universal search engine is the network coverage as big as possible, limited search engine server resource
Contradiction between unlimited network data resource will further deepen；The abundant and network technology of world wide web data form is not
Disconnected development, the different data such as picture, database, audio, video multimedia largely occur, and universal search engine is often to these letters
It is helpless to cease the intensive and data with certain structure of content, cannot find and obtain well；Universal search engine is most
There is provided the retrieval based on keyword, it is difficult to support the inquiry etc. proposed according to semantic information
To solve the above-mentioned problems, the focused crawler of orientation crawl related web page resource comes into being.Focused crawler is one
The program of a automatic downloading webpage, it according to set crawl target, selectively access webpage on WWW to it is relevant
Link, information required for obtaining.Different from general crawler, focused crawler does not pursue big covering, and will be targeted by and grab
Webpage relevant to a certain specific subject content is taken, is greatly saved hardware and Internet resources, the page of preservation is also due to number
Updating decision less is measured, can be good at meeting the needs of some specific crowds are to particular technology area information.
For the scale of current internet, the web crawlers of single machine operation far can not be completed in the effective time
The task of the interior entire WWW of search, therefore the web crawlers used now all distributions are run parallel on multimachine, are claimed
For distributed reptile, however, when the data volume for needing to crawl is huge, existing distributed reptile framework crawl efficiency compared with
It is low.
Summary of the invention
In view of this, the embodiment of the invention provides a kind of distributed vertical crawler method and terminal device, it is existing to solve
Have in technology that distributed reptile crawls the lower problem of efficiency when needing the data volume that crawls huge.
The first aspect of the embodiment of the present invention provides a kind of distributed vertical crawler method, and this method is applied in one kind
Platform is controlled, middle control platform is connected with task distribution platform, Data Analysis Platform and data grabber platform, and this method includes：
The performance data of acquisition task distribution platform, Data Analysis Platform and data grabber platform；
When the value of performance data is more than preset value, warning information is issued.
Further, the performance data of acquisition task distribution platform, Data Analysis Platform and data grabber platform includes：
The performance data of reception task distribution platform, Data Analysis Platform and data grabber platform；Or,
The first instruction is sent to task distribution platform, Data Analysis Platform and data grabber platform according to prefixed time interval
Message, the first instruction message are used to indicate task distribution platform, Data Analysis Platform and data grabber platform and send out performance data
It send to middle control platform；
The performance data of reception task distribution platform, Data Analysis Platform and data grabber platform.
Further, this method further includes：
Crawl task is distributed to data grabber platform by task distribution platform, so that data grabber platform is according to grabbing
Task is taken to carry out the crawl of data.
The second aspect of the embodiment of the present invention provides a kind of distributed vertical crawler method, and this method is applied to a kind of appoint
Be engaged in distribution platform, task distribution platform is connected with middle control platform, Data Analysis Platform and n data grabber platform, wherein n >=
2, this method includes：
The crawl task of platform or Data Analysis Platform is controlled in reception；
According to the network type for grabbing the task type of task, the terminal type of data grabber platform, data grabber platform
At least one of ability is crawled with data grabber platform, determines the distribution policy of crawl task；
According to distribution policy, crawl task is distributed to data grabber platform.
The third aspect of the embodiment of the present invention provides a kind of distributed vertical crawler method, and this method is applied to a kind of number
According to analysis platform, Data Analysis Platform is connected with middle control platform, task distribution platform and data grabber platform, this method packet
It includes：
Receive the crawl result that data grabber platform is sent；
According to crawl as a result, the data extraction strategy that load pre-sets, judges whether there is new crawl task, if so, will
New crawl business is distributed to data grabber platform by task distribution platform, so that data grabber platform is according to new crawl
Task carries out the crawl of data, if not having, crawl result is sent to middle control platform.
Further, the data extraction strategy pre-seted is obtained according to the business scope of crawl task, extraction strategy
It is loaded by way of plug-in unit.
The fourth aspect of the embodiment of the present invention provides a kind of distributed vertical crawler method, and this method is applied to a kind of point
The vertical crawler system of cloth, the system include middle control platform, task distribution platform, Data Analysis Platform, data grabber platform and
Data transfer platform, wherein data transfer platform is used for middle control platform, task distribution platform, Data Analysis Platform and data grabber
Data transmission between platform, this method include：
Crawl task is sent to task distribution platform by middle control platform；
Task distribution platform is flat according to the crawl task type of task, the terminal type of data grabber platform, data grabber
The network type of platform and data grabber platform crawl at least one of ability, determine the distribution policy of crawl task, and root
According to distribution policy, crawl task is distributed to data grabber platform；
Data grabber platform carries out data grabber according to crawl task, and crawl result is sent to Data Analysis Platform；
Data Analysis Platform is according to crawl as a result, the data extraction strategy that load pre-sets, judges whether there is new crawl
Task, if so, new crawl business is distributed to data grabber platform by task distribution platform, so that data grabber platform
Crawl result is sent to middle control platform if not having by the crawl that data are carried out according to new crawl task.
5th aspect of the embodiment of the present invention provides and controls platform in one kind, the middle control platform and task distribution platform,
Data Analysis Platform is connected with data grabber platform, and the middle control platform includes acquiring unit and prewarning unit；
The acquiring unit, for obtaining the performance number of task distribution platform, Data Analysis Platform and data grabber platform
According to；
The prewarning unit, for issuing warning information when the value of the performance data is more than preset value.
6th aspect of the embodiment of the present invention provides a kind of task distribution platform, and the task distribution platform and middle control are flat
Platform, Data Analysis Platform are connected with n data grabber platform, and wherein n is positive integer, n >=2, the task distribution platform packet
Include receiving unit, determination unit and Dispatching Unit；
The receiving unit, for receiving the crawl task of the middle control platform or the Data Analysis Platform；
The determination unit, for the terminal class according to the task type of the crawl task, the data grabber platform
Type, the network type of the data grabber platform and the data grabber platform crawl at least one of ability, determine institute
State the distribution policy of crawl task；
The Dispatching Unit, for the crawl task being distributed to the data grabber and is put down according to the distribution policy
Platform.
7th aspect of the embodiment of the present invention provides a kind of Data Analysis Platform, Data Analysis Platform and middle control platform,
Task distribution platform is connected with data grabber platform, and the Data Analysis Platform includes receiving unit and judging unit；
The receiving unit, the crawl result sent for receiving the data grabber platform；
The judging unit, for judging whether there is according to the data extraction strategy grabbed as a result, load pre-sets
New crawl task is put down if so, the new crawl business is distributed to the data grabber by the task distribution platform
Platform, so that the data grabber platform carries out the crawl of data according to the new crawl task, if not having, by the crawl
As a result it is sent to the middle control platform.
The eighth aspect of the embodiment of the present invention provides and controls platform in one kind, including memory, processor and is stored in
In memory and the computer program that can run on a processor, processor realize the embodiment of the present invention when executing computer program
The step of any one of first aspect method.
9th aspect of the embodiment of the present invention provides a kind of task distribution platform, including memory, processor and deposits
The computer program that can be run in memory and on a processor is stored up, processor is realized when executing computer program such as the present invention
The step of method of embodiment second aspect.
Tenth aspect of the embodiment of the present invention provides a kind of Data Analysis Platform, including memory, processor and deposits
The computer program that can be run in memory and on a processor is stored up, processor is realized when executing computer program such as the present invention
The step of any one of embodiment third aspect method.
Tenth one side of the embodiment of the present invention provides a kind of computer readable storage medium, computer-readable storage medium
Matter is stored with computer program, realizes when computer program is executed by processor such as any one of first aspect of the embodiment of the present invention
The step of method.
12nd aspect of the embodiment of the present invention provides a kind of computer readable storage medium, computer-readable storage medium
Matter is stored with computer program, realizes when computer program is executed by processor such as the method for second aspect of the embodiment of the present invention
Step.
13rd aspect of the embodiment of the present invention provides a kind of computer readable storage medium, computer-readable storage medium
Matter is stored with computer program, realizes when computer program is executed by processor such as any one of third aspect of the embodiment of the present invention
The step of method.
This method is applied to a kind of new distributed vertical crawler system, distributes in the system comprising middle control platform, task
Platform, Data Analysis Platform and data grabber platform, and communicated by data transfer platform.This method includes：Middle control is flat
Crawl task is sent to task distribution platform by platform；Task distribution platform is flat according to task type, the data grabber of crawl task
The terminal type of platform, the network type of data grabber platform and data grabber platform crawl at least one of ability, determine
The distribution policy of crawl task, and according to distribution policy, crawl task is distributed to data grabber platform；Data grabber platform root
Data grabber is carried out according to crawl task, and crawl result is sent to Data Analysis Platform；Data Analysis Platform is tied according to crawl
Fruit loads the data extraction strategy pre-seted, new crawl task is judged whether there is, if so, by new crawl business by appointing
Business distribution platform is distributed to data grabber platform, so that data grabber platform carries out grabbing for data according to new crawl task
It takes, if not having, crawl result is sent to middle control platform.In the method, data grabber platform is only responsible for the crawl of data, when
When the data volume for needing to grab is huge, data grabber efficiency can be improved by this method.
Detailed description of the invention
It to describe the technical solutions in the embodiments of the present invention more clearly, below will be to embodiment or description of the prior art
Needed in attached drawing be briefly described, it should be apparent that, the accompanying drawings in the following description is only of the invention some
Embodiment for those of ordinary skill in the art without any creative labor, can also be according to these
Attached drawing obtains other attached drawings.
Fig. 1 is a kind of distributed vertical crawler framework in the prior art；
Fig. 2 is a kind of implementation process schematic diagram of distributed vertical crawler method provided in an embodiment of the present invention；
Fig. 3 is the implementation process schematic diagram of another distributed vertical crawler method provided in an embodiment of the present invention；
Fig. 4 is the implementation process schematic diagram of another distributed vertical crawler method provided in an embodiment of the present invention；
Fig. 5 is the implementation process schematic diagram of another distributed vertical crawler method provided in an embodiment of the present invention；
Fig. 6 is to control platform schematic diagram in one kind provided in an embodiment of the present invention；
Fig. 7 is a kind of task distribution platform schematic diagram provided in an embodiment of the present invention；
Fig. 8 is a kind of Data Analysis Platform schematic diagram provided in an embodiment of the present invention；
Fig. 9 is to control platform schematic diagram in another kind provided in an embodiment of the present invention；
Figure 10 is another task distribution platform schematic diagram provided in an embodiment of the present invention；
Figure 11 is another Data Analysis Platform schematic diagram provided in an embodiment of the present invention.
Specific embodiment
In being described below, for illustration and not for limitation, the tool of such as particular system structure, technology etc is proposed
Body details, to understand thoroughly the embodiment of the present invention.However, it will be clear to one skilled in the art that there is no these specific
The present invention also may be implemented in the other embodiments of details.In other situations, it omits to well-known system, device, electricity
The detailed description of road and method, in case unnecessary details interferes description of the invention.
In order to illustrate technical solutions according to the invention, the following is a description of specific embodiments.
Fig. 1 is a kind of distributed vertical crawler framework of the prior art, which includes crawler controller and multiple crawlers
End, crawler end is opened from the URL (Uniform Resource Locator, uniform resource locator) of one or several Initial pages
Begin, the URL obtained on Initial page constantly extracts new URL from current page and be put into team during grabbing webpage
Column, certain stop condition until meeting system.The workflow of focused crawler is complex, needs according to certain webpage point
Analysis algorithm filtering is unrelated with theme to be linked, and the URL queue to be captured such as retains useful link and put it into.Then, it
The webpage URL to be grabbed in next step will be selected from queue according to certain search strategy, and is repeated the above process, until reaching
Stop when a certain condition of system.Since the function that crawler end needs to realize is more, when the data volume for needing to crawl is huge, meeting
Cause to crawl efficiency lower.
To solve this problem, the embodiment of the present invention proposes a kind of new distributed vertical crawler system, the system packet
Middle control platform, task distribution platform, Data Analysis Platform, data grabber platform and data transfer platform are included, wherein data are transmitted
Platform is for the data transmission between middle control platform, task distribution platform, Data Analysis Platform and data grabber platform, in conjunction with figure
2, this method includes：
Crawl task is sent to task distribution platform by step S201, middle control platform.
Middle control platform is the mster-control centre of entire distributed vertical crawler system, and middle control platform is flat with task distribution respectively
Platform, Data Analysis Platform are connected, and carry out data communication by data transfer platform and other each platforms.Stability one is
The important parameter of a system is evaluated, middle control platform obtains task distribution platform, Data Analysis Platform by data transfer platform
With the performance data of data grabber platform；When the value of the performance data is more than preset value, warning information is issued.
Optionally, the performance data of platform include but is not limited to memory, CPU (Central Processing Unit, in
Central processor), the correlated performance datas such as hard disk.The property of acquisition task distribution platform, Data Analysis Platform and data grabber platform
The method of energy data includes two kinds：
The first, each platform in system periodically sends its performance number to middle control platform according to the time interval pre-seted
According to middle control platform directly receives the performance data of task distribution platform, Data Analysis Platform and data grabber platform；
Second, middle control platform is according to prefixed time interval to the task distribution platform, Data Analysis Platform and data
It grabs platform and sends the first instruction message, first instruction message is used to indicate the task distribution platform, data analysis is put down
Performance data is sent to the middle control platform by platform and data grabber platform.Task distribution platform, Data Analysis Platform and data
After crawl platform receives the first instruction message, the performance data of itself is sent to middle control platform by data transfer platform,
Middle control platform receives the performance data of the task distribution platform, Data Analysis Platform and data grabber platform.
In addition to this, middle control platform, which provides visualization interface, can obtain other each platform properties by visualization interface
Data can carry out monitoring and early warning to other each platform properties, in order to which administrative staff are effectively to each flat in system
Platform carries out the operation such as dilatation, adjustment of load.
When there is crawl task, crawl task is distributed to data grabber platform by task distribution platform by middle control platform,
So that data grabber platform carries out the crawl of data according to crawl task.
Step S202, task distribution platform is according to the crawl task type of task, the terminal type of data grabber platform, number
At least one of ability is crawled according to the network type and data grabber platform for grabbing platform, determines the distribution plan of crawl task
Slightly, and according to distribution policy, crawl task is distributed to data grabber platform.
Specifically, task type can be divided into link task, picture task, text task dispatching, different task types is not
Priority difference, task amount are different in same application scenarios.
The terminal type of data grabber platform includes computer, server, mobile phone etc., and different terminal capabilities is different, configuration
Also different, the ability for handling crawl task is also different.
The network type of data grabber platform includes as (3rd-Generation, the third generation are mobile logical by the 3G of mobile terminal
Letter technology) network, (the 4th Generation mobile communication technology, forth generation are mobile by 4G
The communication technology) network and 5G (5th-Generation, the 5th third-generation mobile communication technology) network etc., and when access internet
Bandwidth etc..Under different network types data crawl terminal crawl speed difference.
Data grabber platform crawls ability, including the speed crawled, the success rate crawled etc..Rate is crawled to refer to once
It crawls and just crawls several times successfully from reception task to the time spent in crawling completion, crawling success rate and refer to have crawled (because of some quilts
The network crawled has anti-creep mechanism).
For example, such as task distribution platform crawls task for 100 and is distributed to three task crawl platforms, these three
Task crawl platform uses 3G network, 4G network and 5G network respectively, task distribution platform according to data grabber platform network
Type determines distribution policy：Distribute 10 crawl tasks using the data grabber platform of 3G network, is grabbed using the data of 4G network
Platform of making even distributes 35 crawl tasks, distributes 55 crawl tasks using the data grabber terminal of 5G network.
The above is only an examples of the embodiment of the present invention.It is any to be based on inventive concept, according to the task of crawl task
Type, the terminal type of data grabber platform, the network type of data grabber platform and data grabber platform to crawl ability true
Fixed distribution policy, all within the protection scope of the present invention.
Step S203, data grabber platform carries out data grabber according to crawl task, and crawl result is sent to data
Analysis platform.
In embodiments of the present invention, data grabber platform only grabs data, the analysis and screening without data
Deng ensure that the grasp speed of data grabber platform.
Step S204, Data Analysis Platform is according to crawl as a result, the data extraction strategy that load pre-sets, judges whether there is
New crawl task, if so, new crawl business is distributed to data grabber platform by task distribution platform, so that data
The crawl that platform carries out data according to new crawl task is grabbed, if not having, crawl result is sent to middle control platform.
Specifically, if Data Analysis Platform according to crawl as a result, judgement have new crawl task, new crawl business is led to
The task distribution platform of mistake is distributed to data grabber platform, so that data grabber platform carries out data according to new crawl task
Final crawl result is sent to middle control platform until Data Analysis Platform judges not new crawl task by crawl.
Further, in conjunction with Fig. 3, the embodiment of the invention also provides a kind of distributed vertical crawler method, this method is answered
For controlling platform in one kind, middle control platform is connected with task distribution platform, Data Analysis Platform and data grabber platform, the party
Method includes：
S301 obtains the performance data of task distribution platform, Data Analysis Platform and data grabber platform.
Optionally, the performance data of platform include but is not limited to memory, CPU (Central Processing Unit, in
Central processor), the correlated performance datas such as hard disk.
Preferably, the method packet of the performance data of task distribution platform, Data Analysis Platform and data grabber platform is obtained
Include two kinds：
The first, each platform in system periodically sends its performance number to middle control platform according to the time interval pre-seted
According to middle control platform directly receives the performance data of task distribution platform, Data Analysis Platform and data grabber platform；
Second, middle control platform is according to prefixed time interval to the task distribution platform, Data Analysis Platform and data
It grabs platform and sends the first instruction message, first instruction message is used to indicate the task distribution platform, data analysis is put down
Performance data is sent to the middle control platform by platform and data grabber platform.Task distribution platform, Data Analysis Platform and data
After crawl platform receives the first instruction message, the performance data of itself is sent to middle control platform by data transfer platform,
Middle control platform receives the performance data of the task distribution platform, Data Analysis Platform and data grabber platform.
S302 issues warning information when the value of the performance data is more than preset value.
Crawl task is distributed to the data grabber platform by the task distribution platform, so that described by S303
Data grabber platform carries out the crawl of data according to the crawl task.
Further, in conjunction with Fig. 4, the embodiment of the invention also provides a kind of distributed vertical crawler method, this method is answered
For a kind of task distribution platform, the task distribution platform and middle control platform, Data Analysis Platform and n data grabber platform
It is connected, wherein n is positive integer, and n >=2, this method includes：
S401 receives the crawl task of the middle control platform or the Data Analysis Platform.
S402 is grabbed according to the task type of the crawl task, the terminal type of the data grabber platform, the data
It makes even the network type of platform and the data grabber platform crawls at least one of ability, determine point of the crawl task
Hair strategy.
The crawl task is distributed to the data grabber platform according to the distribution policy by S403.
Further, in conjunction with Fig. 5, the embodiment of the invention also provides a kind of distributed vertical crawler method, this method is answered
For a kind of Data Analysis Platform, the Data Analysis Platform and middle control platform, task distribution platform and data grabber platform phase
Connection, this method include：
S501 receives the crawl result that the data grabber platform is sent.
S502 judges whether there is new crawl and appoints according to the crawl as a result, the data that load pre-sets extract strategy
Business, if so, the new crawl business is distributed to the data grabber platform by the task distribution platform, so that institute
Data grabber platform is stated to be sent to the crawl result if not having according to the crawl that the new crawl task carries out data
The middle control platform.
Preferably, the data extraction strategy pre-seted is obtained according to the business scope of the crawl task, institute
Extraction strategy is stated to be loaded by way of plug-in unit.
The embodiment of the invention provides a kind of method of distributed vertical crawler, this method is applied to a kind of new distribution
Vertical crawler system, includes middle control platform, task distribution platform, Data Analysis Platform and data grabber platform in the system, and
It is communicated by data transfer platform.This method includes：Crawl task is sent to task distribution platform by middle control platform；Task
Distribution platform is according to the network type for grabbing the task type of task, the terminal type of data grabber platform, data grabber platform
At least one of ability is crawled with data grabber platform, determines the distribution policy of crawl task, and according to distribution policy, it will
Crawl task is distributed to data grabber platform；Data grabber platform carries out data grabber according to crawl task, and will grab result
It is sent to Data Analysis Platform；Data Analysis Platform is according to crawl as a result, the data extraction strategy that load pre-sets, judges whether
There is new crawl task, if so, new crawl business is distributed to data grabber platform by task distribution platform, so that total
Crawl result is sent to middle control platform if not having by the crawl for carrying out data according to new crawl task according to crawl platform.At this
In method, data grabber platform is only responsible for the crawl of data, when the data volume for needing to grab is huge, can be improved by this method
Data grabber efficiency.
It should be understood that the size of the serial number of each step is not meant that the order of the execution order in above-described embodiment, each process
Execution sequence should be determined by its function and internal logic, the implementation process without coping with the embodiment of the present invention constitutes any limit
It is fixed.
Fig. 6 is to control platform schematic diagram in one kind provided in an embodiment of the present invention, as shown in fig. 6, middle control platform and task point
Hair platform, Data Analysis Platform are connected with data grabber platform, and the middle control platform includes acquiring unit 61 and prewarning unit
62；
The acquiring unit 61, for obtaining the performance of task distribution platform, Data Analysis Platform and data grabber platform
Data；
The prewarning unit 62, for issuing warning information when the value of the performance data is more than preset value.
Preferably, acquiring unit 61 is specifically used for：
Receive the performance data of the task distribution platform, Data Analysis Platform and data grabber platform；Or,
First is sent to the task distribution platform, Data Analysis Platform and data grabber platform according to prefixed time interval
Instruction message, first instruction message are used to indicate the task distribution platform, Data Analysis Platform and data grabber platform
Performance data is sent to the middle control platform；
Receive the performance data of the task distribution platform, Data Analysis Platform and data grabber platform.
Further, middle control platform 6 further includes Dispatching Unit 63, passes through the task distribution platform for that will grab task
It is distributed to the data grabber platform, so that the data grabber platform carries out the crawl of data according to the crawl task.
Further, in conjunction with Fig. 7, the embodiment of the invention also provides a kind of task distribution platform, the task distribution is flat
Platform is connected with middle control platform, Data Analysis Platform and n data grabber platform, and wherein n is positive integer, n >=2, the task
Distribution platform includes receiving unit 71, determination unit 72 and Dispatching Unit 73；
The receiving unit 71, for receiving the crawl task of the middle control platform or the Data Analysis Platform；
The determination unit 72, for the terminal according to the task type of the crawl task, the data grabber platform
Type, the network type of the data grabber platform and the data grabber platform crawl at least one of ability, determine
The distribution policy of the crawl task；
The Dispatching Unit 73, for according to the distribution policy, the crawl task to be distributed to the data grabber
Platform.
Further, in conjunction with Fig. 8, the embodiment of the invention also provides a kind of Data Analysis Platform, Data Analysis Platform with
Middle control platform, task distribution platform are connected with data grabber platform, and the Data Analysis Platform includes receiving unit 81 and sentences
Disconnected unit 82；
The receiving unit 81, the crawl result sent for receiving the data grabber platform；
The judging unit 82, for judging whether according to the data extraction strategy grabbed as a result, load pre-sets
There is new crawl task, if so, the new crawl business is distributed to the data grabber by the task distribution platform
Platform, so that the data grabber platform is grabbed if not having by described according to the crawl that the new crawl task carries out data
Result is taken to be sent to the middle control platform.
Preferably, the data extraction strategy pre-seted is obtained according to the business scope of the crawl task, institute
Extraction strategy is stated to be loaded by way of plug-in unit.
The embodiment of the invention provides a kind of distributed vertical crawler system, which includes middle control platform, task distribution
Platform, Data Analysis Platform and data grabber platform, each platform are communicated by data transfer platform.Middle control platform will grab
Task is sent to task distribution platform；Task distribution platform is according to the task type of crawl task, the terminal of data grabber platform
Type, the network type of data grabber platform and data grabber platform crawl at least one of ability, determine crawl task
Distribution policy crawl task is distributed to data grabber platform and according to distribution policy；Data grabber platform is appointed according to crawl
Business carries out data grabber, and crawl result is sent to Data Analysis Platform；Data Analysis Platform is according to crawl as a result, load is pre-
The data of setting extract strategy, judge whether there is new crawl task, if so, new crawl business is passed through task distribution platform
It is distributed to data grabber platform, so that data grabber platform carries out the crawl of data according to new crawl task, it, will if not having
Crawl result is sent to middle control platform.Within the system, data grabber platform is only responsible for the crawl of data, when the number that needs grab
When according to measuring huge, data grabber efficiency can be improved by this method.
Fig. 9 is the schematic diagram of middle control platform provided in an embodiment of the present invention.As shown in figure 9, the middle control platform 9 of the embodiment
Including：Processor 90, memory 91 and it is stored in the calculating that can be run in the memory 91 and on the processor 90
Machine program 92, such as a kind of program of distributed vertical crawler method.When the processor 90 executes the computer program 92
Realize it is above-mentioned it is each it is middle control platform distributed vertical crawler embodiment of the method in step, such as step 301 shown in Fig. 3 to
303 part, alternatively, the processor 90 realizes each unit in above-mentioned each Installation practice when executing the computer program 92
Function, such as the function of module 61 to 63 shown in Fig. 6.
Illustratively, the computer program 92 can be divided into one or more module/units, it is one or
Multiple module/units are stored in the memory 91, and are executed by the processor 90, to complete the present invention.Described one
A or multiple module/units can be the series of computation machine program instruction section that can complete specific function, which is used for
Implementation procedure of the computer program 92 in the middle control platform 9 is described.For example, the computer program 92 can be divided
It is cut into synchronization module, summarizing module, obtains module, return module (module in virtual bench).
The middle control platform 9 can be the calculating such as desktop PC, notebook, palm PC and cloud server and set
It is standby.The middle control platform 9 may include, but be not limited only to, processor 90, memory 91.It will be understood by those skilled in the art that figure
9 be only the example of middle control platform 9, does not constitute the restriction of centering control platform 9, may include than illustrating more or fewer portions
Part perhaps combines certain components or different components, such as the middle control platform 9 can also include input-output equipment, net
Network access device, bus etc..
Alleged processor 90 can be central processing unit (Central Processing Unit, CPU), can also be
Other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor
Deng.
The memory 91 can be the internal storage unit of the middle control platform 9, such as the hard disk or interior of middle control platform 9
It deposits.The memory 91 is also possible to be equipped on the External memory equipment of the middle control platform 9, such as the middle control platform 9
Plug-in type hard disk, intelligent memory card (Smart Media Card, SMC), secure digital (Secure Digital, SD) card dodge
Deposit card (Flash Card) etc..Further, the memory 91 can also both include the storage inside list of the middle control platform 9
Member also includes External memory equipment.The memory 91 is for storing needed for the computer program and the middle control platform 9
Other programs and data.The memory 91 can be also used for temporarily storing the data that has exported or will export.
Figure 10 is the schematic diagram of task distribution platform provided in an embodiment of the present invention.As shown in Figure 10, times of the embodiment
Business distribution platform 10 include：It processor 100, memory 101 and is stored in the memory 101 and can be in the processor
The computer program 102 run on 100, such as a kind of program of distributed vertical crawler method.The processor 100 executes institute
The step in the distributed vertical crawler embodiment of the method for above-mentioned each task distribution platform is realized when stating computer program 102,
Such as step 401 shown in Fig. 4 is to 403 part.Alternatively, realization when the processor 100 executes the computer program 102
The function of each unit in above-mentioned each Installation practice, such as the function of module 71 to 73 shown in Fig. 7.Illustratively, the calculating
Machine program 102 can be divided into one or more module/units, and one or more of module/units are stored in institute
It states in memory 101, and is executed by the processor 100, to complete the present invention.One or more of module/units can be with
It is the series of computation machine program instruction section that can complete specific function, the instruction segment is for describing the computer program 102
Implementation procedure in the task distribution platform 10.For example, the computer program 102 can be divided into synchronization module,
Summarizing module obtains module, return module (module in virtual bench).
The task distribution platform 10 can be the meter such as desktop PC, notebook, palm PC and cloud server
Calculate equipment.The task distribution platform 10 may include, but be not limited only to, processor 100, memory 101.Those skilled in the art
It is appreciated that Figure 10 is only the example of task distribution platform 10, the restriction to task distribution platform 10 is not constituted, can wrap
It includes than illustrating more or fewer components, perhaps combines certain components or different components, such as the task distribution platform
10 can also include input-output equipment, network access equipment, bus etc..
Alleged processor 100 can be central processing unit (Central Processing Unit, CPU), can also be
Other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor
Deng.
The memory 101 can be the internal storage unit of the task distribution platform 10, such as task distribution platform
10 hard disk or memory.The memory 101 is also possible to the External memory equipment of the task distribution platform 10, such as described
The plug-in type hard disk being equipped on task distribution platform 10, intelligent memory card (Smart Media Card, SMC), secure digital
(Secure Digital, SD) card, flash card (Flash Card) etc..Further, the memory 101 can also both include
The internal storage unit of the task distribution platform 10 also includes External memory equipment.The memory 101 is described for storing
Other programs and data needed for computer program and the task distribution platform 10.The memory 101 can be also used for
Temporarily store the data that has exported or will export.
Figure 11 is the schematic diagram of Data Analysis Platform provided in an embodiment of the present invention.As shown in figure 11, the number of the embodiment
Include according to analysis platform 11：It processor 110, memory 111 and is stored in the memory 111 and can be in the processor
The computer program 112 run on 110, such as a kind of program of distributed vertical crawler method.The processor 110 executes institute
The step in the distributed vertical crawler embodiment of the method for above-mentioned each Data Analysis Platform is realized when stating computer program 112,
Such as step 501 shown in fig. 5 is to 502 part.Alternatively, realization when the processor 110 executes the computer program 112
The function of each unit in above-mentioned each Installation practice, such as the function of module 81 to 82 shown in Fig. 8.
Illustratively, the computer program 112 can be divided into one or more module/units, it is one or
Multiple module/the units of person are stored in the memory 111, and are executed by the processor 110, to complete the present invention.Institute
Stating one or more module/units can be the series of computation machine program instruction section that can complete specific function, the instruction segment
For describing implementation procedure of the computer program 112 in the Data Analysis Platform 11.For example, the computer program
112 can be divided into synchronization module, summarizing module, obtain module, return module (module in virtual bench).
The Data Analysis Platform 11 can be the meter such as desktop PC, notebook, palm PC and cloud server
Calculate equipment.The Data Analysis Platform 11 may include, but be not limited only to, processor 110, memory 111.Those skilled in the art
It is appreciated that Figure 11 is only the example of Data Analysis Platform 11, the restriction of structure paired data analysis platform 11, not can wrap
It includes than illustrating more or fewer components, perhaps combines certain components or different components, such as the Data Analysis Platform
11 can also include input-output equipment, network access equipment, bus etc..
Alleged processor 110 can be central processing unit (Central Processing Unit, CPU), can also be
Other general processors, digital signal processor (Digital Signal Processor, DSP), specific integrated circuit
(Application Specific Integrated Circuit, ASIC), ready-made programmable gate array (Field-
Programmable Gate Array, FPGA) either other programmable logic device, discrete gate or transistor logic,
Discrete hardware components etc..General processor can be microprocessor or the processor is also possible to any conventional processor
Deng.
The memory 111 can be the internal storage unit of the Data Analysis Platform 11, such as Data Analysis Platform
11 hard disk or memory.The memory 111 is also possible to the External memory equipment of the Data Analysis Platform 11, such as described
The plug-in type hard disk being equipped on Data Analysis Platform 11, intelligent memory card (Smart Media Card, SMC), secure digital
(Secure Digital, SD) card, flash card (Flash Card) etc..Further, the memory 111 can also both include
The internal storage unit of the Data Analysis Platform 11 also includes External memory equipment.The memory 111 is described for storing
Other programs and data needed for computer program and the Data Analysis Platform 11.The memory 111 can be also used for
Temporarily store the data that has exported or will export.
It is apparent to those skilled in the art that for convenience of description and succinctly, only with above-mentioned each function
Can unit, module division progress for example, in practical application, can according to need and by above-mentioned function distribution by different
Functional unit, module are completed, i.e., the internal structure of described device is divided into different functional unit or module, more than completing
The all or part of function of description.Each functional unit in embodiment, module can integrate in one processing unit, can also
To be that each unit physically exists alone, can also be integrated in one unit with two or more units, it is above-mentioned integrated
Unit both can take the form of hardware realization, can also realize in the form of software functional units.In addition, each function list
Member, the specific name of module are also only for convenience of distinguishing each other, the protection scope being not intended to limit this application.Above system
The specific work process of middle unit, module, can refer to corresponding processes in the foregoing method embodiment, and details are not described herein.
In the above-described embodiments, it all emphasizes particularly on different fields to the description of each embodiment, is not described in detail or remembers in some embodiment
The part of load may refer to the associated description of other embodiments.
Those of ordinary skill in the art may be aware that list described in conjunction with the examples disclosed in the embodiments of the present disclosure
Member and algorithm steps can be realized with the combination of electronic hardware or computer software and electronic hardware.These functions are actually
It is implemented in hardware or software, the specific application and design constraint depending on technical solution.Professional technician
Each specific application can be used different methods to achieve the described function, but this realization is it is not considered that exceed
The scope of the present invention.
In embodiment provided by the present invention, it should be understood that disclosed platform and method can pass through others
Mode is realized.For example, the embodiment of a platform described above is only schematical, for example, the module or unit
It divides, only a kind of logical function partition, there may be another division manner in actual implementation, such as multiple units or components
It can be combined or can be integrated into another system, or some features can be ignored or not executed.Another point, it is shown or
The mutual coupling or direct-coupling or communication connection discussed can be through some interfaces, the indirect coupling of device or unit
Conjunction or communication connection can be electrical property, mechanical or other forms.
The unit as illustrated by the separation member may or may not be physically separated, aobvious as unit
The component shown may or may not be physical unit, it can and it is in one place, or may be distributed over multiple
In network unit.It can select some or all of unit therein according to the actual needs to realize the mesh of this embodiment scheme
's.
It, can also be in addition, the functional units in various embodiments of the present invention may be integrated into one processing unit
It is that each unit physically exists alone, can also be integrated in one unit with two or more units.Above-mentioned integrated list
Member both can take the form of hardware realization, can also realize in the form of software functional units.
If the integrated module/unit be realized in the form of SFU software functional unit and as independent product sale or
In use, can store in a computer readable storage medium.Based on this understanding, the present invention realizes above-mentioned implementation
All or part of the process in example method, can also instruct relevant hardware to complete, the meter by computer program
Calculation machine program can be stored in computer readable storage medium, and the computer program is when being executed by processor, it can be achieved that above-mentioned
The step of each embodiment of the method.Wherein, the computer program includes computer program code, the computer program code
It can be source code form, object identification code form, executable file or certain intermediate forms etc..The computer-readable medium can
To include：Can carry the computer program code any entity or device, recording medium, USB flash disk, mobile hard disk, magnetic disk,
CD, computer storage, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random
Access Memory), electric carrier signal, telecommunication signal and software distribution medium etc..It should be noted that the computer
The content that readable medium includes can carry out increase and decrease appropriate according to the requirement made laws in jurisdiction with patent practice, such as
It does not include electric carrier signal and telecommunication signal according to legislation and patent practice, computer-readable medium in certain jurisdictions.
Embodiment described above is merely illustrative of the technical solution of the present invention, rather than its limitations；Although referring to aforementioned reality
Applying example, invention is explained in detail, those skilled in the art should understand that：It still can be to aforementioned each
Technical solution documented by embodiment is modified or equivalent replacement of some of the technical features；And these are modified
Or replacement, the spirit and scope for technical solution of various embodiments of the present invention that it does not separate the essence of the corresponding technical solution should all
It is included within protection scope of the present invention.
Claims (16)
1. a kind of distributed vertical crawler method, which is characterized in that this method is applied to control platform in one kind, middle control platform with times
Business distribution platform, Data Analysis Platform are connected with data grabber platform, and this method includes：
The performance data of acquisition task distribution platform, Data Analysis Platform and data grabber platform；
When the value of the performance data is more than preset value, warning information is issued.
2. the method according to claim 1, wherein obtaining task distribution platform, Data Analysis Platform and data
Crawl platform performance data include：
Receive the performance data of the task distribution platform, Data Analysis Platform and data grabber platform；Or,
The first instruction is sent to the task distribution platform, Data Analysis Platform and data grabber platform according to prefixed time interval
Message, first instruction message are used to indicate the task distribution platform, Data Analysis Platform and data grabber platform for property
Energy data are sent to the middle control platform；
Receive the performance data of the task distribution platform, Data Analysis Platform and data grabber platform.
3. method according to claim 1 or 2, which is characterized in that this method further includes：
Crawl task is distributed to the data grabber platform by the task distribution platform, so that the data grabber is flat
Platform carries out the crawl of data according to the crawl task.
4. a kind of distributed vertical crawler method, which is characterized in that this method is applied to a kind of task distribution platform, the task
Distribution platform is connected with middle control platform, Data Analysis Platform and n data grabber platform, and wherein n is positive integer, and n >=2 should
Method includes：
Receive the crawl task of the middle control platform or the Data Analysis Platform；
According to the task type of the crawl task, the terminal type of the data grabber platform, the data grabber platform
Network type and the data grabber platform crawl at least one of ability, determine the distribution policy of the crawl task；
According to the distribution policy, the crawl task is distributed to the data grabber platform.
5. a kind of distributed vertical crawler method, which is characterized in that this method is applied to a kind of Data Analysis Platform, the data
Analysis platform is connected with middle control platform, task distribution platform and data grabber platform, and this method includes：
Receive the crawl result that the data grabber platform is sent；
According to the crawl as a result, the data extraction strategy that load pre-sets, judges whether there is new crawl task, if so, will
The new crawl business is distributed to the data grabber platform by the task distribution platform, so that the data grabber
Platform carries out the crawl of data according to the new crawl task, if not having, it is flat that the crawl result is sent to the middle control
Platform.
6. according to the method described in claim 5, it is characterized in that, the data extraction strategy pre-seted is grabbed according to
The business scope of task is taken to obtain, the extraction strategy is loaded by way of plug-in unit.
7. a kind of distributed vertical crawler method, which is characterized in that this method is applied to a kind of distributed vertical crawler system, should
System includes middle control platform, task distribution platform, Data Analysis Platform, data grabber platform and data transfer platform, wherein institute
Data transfer platform is stated between the middle control platform, task distribution platform, Data Analysis Platform and data grabber platform
Data transmission, this method include：
Crawl task is sent to the task distribution platform by the middle control platform；
The task distribution platform is according to the task type of the crawl task, the terminal type of the data grabber platform, institute
The network type and the data grabber platform for stating data grabber platform crawl at least one of ability, determine the crawl
The distribution policy of task, and according to the distribution policy, the crawl task is distributed to the data grabber platform；
The data grabber platform carries out data grabber according to the crawl task, and crawl result is sent to the data point
Analyse platform；
The Data Analysis Platform is according to the crawl as a result, the data extraction strategy that load pre-sets, judges whether there is new
Crawl task, if so, the new crawl business is distributed to the data grabber platform by the task distribution platform, with
So that the data grabber platform carries out the crawl of data according to the new crawl task, if not having, by the crawl result
It is sent to the middle control platform.
8. controlling platform in one kind, which is characterized in that the middle control platform is grabbed with task distribution platform, Data Analysis Platform and data
Platform of making even is connected, and the middle control platform includes acquiring unit and prewarning unit；
The acquiring unit, for obtaining the performance data of task distribution platform, Data Analysis Platform and data grabber platform；
The prewarning unit, for issuing warning information when the value of the performance data is more than preset value.
9. a kind of task distribution platform, which is characterized in that the task distribution platform and middle control platform, Data Analysis Platform and n
A data grabber platform is connected, and wherein n is positive integer, and n >=2, the task distribution platform includes receiving unit, determination unit
And Dispatching Unit；
The receiving unit, for receiving the crawl task of the middle control platform or the Data Analysis Platform；
The determination unit, for according to the task type of the crawl task, the terminal type of the data grabber platform, institute
The network type and the data grabber platform for stating data grabber platform crawl at least one of ability, determine the crawl
The distribution policy of task；
The Dispatching Unit, for according to the distribution policy, the crawl task to be distributed to the data grabber platform.
10. a kind of Data Analysis Platform, which is characterized in that the Data Analysis Platform and middle control platform, task distribution platform and
Data grabber platform is connected, and the Data Analysis Platform includes receiving unit and judging unit；
The receiving unit, the crawl result sent for receiving the data grabber platform；
The judging unit, for judging whether there is new according to the data extraction strategy grabbed as a result, load pre-sets
Crawl task, if so, the new crawl business is distributed to the data grabber platform by the task distribution platform, with
So that the data grabber platform carries out the crawl of data according to the new crawl task, if not having, by the crawl result
It is sent to the middle control platform.
11. control platform in one kind, including memory, processor and storage are in the memory and can be on the processor
The computer program of operation, which is characterized in that the processor realizes such as claims 1 to 3 when executing the computer program
The step of any one the method.
12. a kind of task distribution platform, including memory, processor and storage are in the memory and can be in the processing
The computer program run on device, which is characterized in that the processor realizes such as claim 4 when executing the computer program
The step of the method.
13. a kind of Data Analysis Platform, including memory, processor and storage are in the memory and can be in the processing
The computer program run on device, which is characterized in that the processor realizes such as claim 5 when executing the computer program
The step of to any one of 6 the method.
14. a kind of computer readable storage medium, the computer-readable recording medium storage has computer program, and feature exists
In when the computer program is executed by processor the step of any one of such as claims 1 to 3 of realization the method.
15. a kind of computer readable storage medium, the computer-readable recording medium storage has computer program, and feature exists
In the step of computer program realizes method as claimed in claim 4 when being executed by processor.
16. a kind of computer readable storage medium, the computer-readable recording medium storage has computer program, and feature exists
In when the computer program is executed by processor the step of any one of such as claim 5 to 6 of realization the method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810547735.4A CN108874925A (en) | 2018-05-31 | 2018-05-31 | A kind of distributed vertical crawler method and terminal device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810547735.4A CN108874925A (en) | 2018-05-31 | 2018-05-31 | A kind of distributed vertical crawler method and terminal device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN108874925A true CN108874925A (en) | 2018-11-23 |
Family
ID=64336191
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810547735.4A Pending CN108874925A (en) | 2018-05-31 | 2018-05-31 | A kind of distributed vertical crawler method and terminal device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN108874925A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111026945A (en) * | 2019-12-05 | 2020-04-17 | 北京创鑫旅程网络技术有限公司 | Multi-platform crawler scheduling method and device and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1145158A2 (en) * | 1999-01-27 | 2001-10-17 | On Guard Plus Limited | System for data capture, normalization, data event processing, communication and operator interface |
| CN103037010A (en) * | 2012-12-26 | 2013-04-10 | 人民搜索网络股份公司 | Distributed network crawler system and catching method thereof |
| CN104951512A (en) * | 2015-05-27 | 2015-09-30 | 中国科学院信息工程研究所 | Public sentiment data collection method and system based on Internet |
| CN105447088A (en) * | 2015-11-06 | 2016-03-30 | 杭州掘数科技有限公司 | Volunteer computing based multi-tenant professional cloud crawler |
| CN105989151A (en) * | 2015-03-02 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Webpage crawling method and apparatus |
| CN107180050A (en) * | 2016-03-11 | 2017-09-19 | 精硕科技（北京）股份有限公司 | A kind of data grabber system and method |
-
2018
- 2018-05-31 CN CN201810547735.4A patent/CN108874925A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP1145158A2 (en) * | 1999-01-27 | 2001-10-17 | On Guard Plus Limited | System for data capture, normalization, data event processing, communication and operator interface |
| CN103037010A (en) * | 2012-12-26 | 2013-04-10 | 人民搜索网络股份公司 | Distributed network crawler system and catching method thereof |
| CN105989151A (en) * | 2015-03-02 | 2016-10-05 | 阿里巴巴集团控股有限公司 | Webpage crawling method and apparatus |
| CN104951512A (en) * | 2015-05-27 | 2015-09-30 | 中国科学院信息工程研究所 | Public sentiment data collection method and system based on Internet |
| CN105447088A (en) * | 2015-11-06 | 2016-03-30 | 杭州掘数科技有限公司 | Volunteer computing based multi-tenant professional cloud crawler |
| CN107180050A (en) * | 2016-03-11 | 2017-09-19 | 精硕科技（北京）股份有限公司 | A kind of data grabber system and method |
Non-Patent Citations (4)
| Title |
|---|
| 刘庆 等: "《数据库系统概论》", 30 September 2015 * |
| 杨旭 等: "《供港食品全程溯源与实时监控关键技术及其应用》", 31 January 2017 * |
| 特性: "《网络内容管理与情报分析》", 30 June 2009 * |
| 邓宁: "《全国目的地网络舆情监测报告》", 30 June 2016 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111026945A (en) * | 2019-12-05 | 2020-04-17 | 北京创鑫旅程网络技术有限公司 | Multi-platform crawler scheduling method and device and storage medium |
| CN111026945B (en) * | 2019-12-05 | 2024-01-26 | 北京创鑫旅程网络技术有限公司 | Multi-platform crawler scheduling method, device and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110535831A (en) | Cluster safety management method, device and storage medium based on Kubernetes and network domains | |
| CN109815657A (en) | A kind of identity identifying method and terminal device based on alliance's chain | |
| CN109816321A (en) | A kind of service management, device, equipment and computer readable storage medium | |
| CN109634728A (en) | Job scheduling method, device, terminal device and readable storage medium storing program for executing | |
| CN101973031A (en) | Cloud robot system and implementation method | |
| CN107438833A (en) | A kind of data-updating method, device, system and server | |
| CN110175027A (en) | A kind of method and apparatus for developing business function | |
| CN110134711A (en) | Processing method, device, equipment and the computer readable storage medium of big data | |
| CN110221145A (en) | Fault Diagnosis for Electrical Equipment method, apparatus and terminal device | |
| CN109614539A (en) | Data grab method, device and computer readable storage medium | |
| CN108255936A (en) | A kind of edit methods of webpage, system and editing machine | |
| CN205845090U (en) | Electricity market main body credit evaluation system | |
| CN107180050A (en) | A kind of data grabber system and method | |
| CN109145188A (en) | For searching for the method, equipment and computer readable storage medium of block chain data | |
| CN108959565A (en) | A kind of method, apparatus and server of web page contents filtering | |
| CN110147397A (en) | System docking method, apparatus, management system and terminal device, storage medium | |
| CN107609797A (en) | Electric operating checking method and terminal device | |
| CN107807935A (en) | Using recommendation method and device | |
| CN108874925A (en) | A kind of distributed vertical crawler method and terminal device | |
| CN109213742A (en) | Log collection method and device | |
| CN110532448B (en) | Document classification method, device, equipment and storage medium based on neural network | |
| CN106844467A (en) | Method for exhibiting data and device | |
| CN110019501A (en) | A kind of collecting method, device and terminal device | |
| CN110247818A (en) | A kind of data monitoring method, device, storage medium and server | |
| CN110442753A (en) | A kind of chart database auto-creating method and device based on OPC UA |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20181123 |
|
| RJ01 | Rejection of invention patent application after publication |