CN107103079B - Live broadcast method and system for dynamic website - Google Patents
Live broadcast method and system for dynamic website Download PDFInfo
- Publication number
- CN107103079B CN107103079B CN201710278347.6A CN201710278347A CN107103079B CN 107103079 B CN107103079 B CN 107103079B CN 201710278347 A CN201710278347 A CN 201710278347A CN 107103079 B CN107103079 B CN 107103079B
- Authority
- CN
- China
- Prior art keywords
- dynamic website
- latest data
- preset
- user
- authorization information
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/23—Updating
- G06F16/2358—Change logging, detection, and notification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
- G06F16/252—Integrating or interfacing systems involving database management systems between a Database Management System and a front-end application
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/85—Assembly of content; Generation of multimedia applications
- H04N21/854—Content authoring
- H04N21/8543—Content authoring using a description language, e.g. Multimedia and Hypermedia information coding Expert Group [MHEG], eXtensible Markup Language [XML]
Landscapes
- Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Security & Cryptography (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a live broadcast method and a system of a dynamic website, wherein the method comprises the following steps: obtaining a token which is permanently logged in the dynamic website and corresponds to the authorization information according to the authorization information of the selected user in the dynamic website; allocating a crawler robot to each token, acquiring current latest data of a preset user of the dynamic website by using the crawler robot, and storing the current latest data into a database; monitoring current latest data stored in a database, acquiring the latest data updated by a preset user, and sending the latest data to a preset live broadcast platform; according to the method and the system, the access limit of the dynamic website to the crawler robots is solved by obtaining the tokens for permanently logging in the dynamic website, the current latest data of the preset user of the dynamic website can be captured by using a plurality of crawler robots by allocating one crawler robot to each token, the limitation of the dynamic website such as the Xinlang microblog on the crawling frequency is effectively avoided, and the real-time performance of live broadcast is improved.
Description
Technical Field
The invention relates to the technical field of computer application, in particular to a live broadcast method and system of a dynamic website.
Background
With the development of modern society science and technology, people have higher and higher requirements on life, and Crawl Robot is also required to be applied to various aspects of life, such as a live broadcast system. A Crawl Robot is also called a crawler Robot, a web spider or a web Robot, downloads web pages from the Internet for a search engine, collects resources from related web page links, and is a powerful automatic web page grabbing program. At the beginning of the birth of the technology, the technology is mainly applied to a search engine and is an important component for determining the search performance and the expansibility of the search engine.
In the prior art, most live broadcast systems are usually oriented to static websites, and are oriented to login-free websites. With the increasing security of websites in the current stage, for example, dynamic websites such as the Sina microblog, a user name, a password and an authentication code are often required to be provided when logging in the website, so that the conventional live broadcasting system cannot utilize the conventional crawler robot to broadcast relevant information to the dynamic websites in a live broadcasting manner. In addition, the Sing microblog monitors related requests, and the requests from a certain IP in a certain period of time are too frequent to deny access to the IP for a certain period of time. Therefore, the traditional live broadcast system cannot carry out live broadcast on dynamic websites such as the Sina microblog.
This results in the need for the relevant personnel to obtain the latest information, which requires continuous refreshing of the page to obtain the target data, however, this operation not only consumes much effort, but also makes it difficult to avoid the intolerable delay. For example, a large number of V in a new wave microblog and a blog can live stock market quotations and related stock information in real time, and people can acquire related information only by refreshing a page in the past, so that the defect that not only manpower is consumed, but also key important information is missed is caused. Therefore, how to live the data in the dynamic website by using the crawler robot reduces the live broadcast delay time and improves the user experience is a problem which needs to be solved urgently nowadays.
Disclosure of Invention
The invention aims to provide a live broadcast method and a live broadcast system for a dynamic website, which are used for live broadcast of data in the dynamic website by using a crawler robot, so that the live broadcast delay time is reduced, and the user experience is improved.
In order to solve the technical problem, the invention provides a live broadcast method of a dynamic website, which comprises the following steps:
obtaining a token which is corresponding to the authorization information and is permanently logged in the dynamic website according to the authorization information of a selected user in the dynamic website; the authorization information comprises login information of a corresponding selected user for logging in the dynamic website;
allocating a crawler robot to each token, acquiring current latest data of a preset user of the dynamic website by using the crawler robot, and storing the current latest data into a database;
and monitoring the current latest data stored in the database, acquiring the latest data updated by the preset user, and sending the latest data to a preset live broadcast platform.
Optionally, the obtaining, according to the authorization information of the selected user in the dynamic website, the token that permanently logs in the dynamic website and corresponds to the authorization information includes:
and according to the authorization information, simulating a user to obtain a token which is corresponding to the authorization information and permanently logs in the dynamic website through an Oauth2.0 protocol.
Optionally, before the simulating, according to the authorization information and through the oauth2.0 protocol, that the user obtains the token permanently logging in the dynamic website corresponding to the authorization information, the method further includes:
sending an authorization request for obtaining the authorization information to the selected user;
and acquiring the authorization information returned by the selected user.
Optionally, the obtaining, by using the crawler robot, current latest data of a preset user of the dynamic website includes:
distributing a preset number of the crawler robots to each preset user;
and each crawler robot corresponding to the preset user sequentially acquires the current latest data within respective preset time.
Optionally, the method further includes:
and counting all the latest data corresponding to each preset user according to preset indexes through a fixed platform website and displaying a counting result.
In addition, the invention also provides a live broadcast system of the dynamic website, which comprises the following steps:
the access authorization module is used for acquiring a token which is corresponding to the authorization information and is used for permanently logging in the dynamic website according to the authorization information of the selected user in the dynamic website; the authorization information comprises login information of a corresponding selected user for logging in the dynamic website;
the data acquisition module is used for allocating a crawler robot to each token, acquiring the current latest data of a preset user of the dynamic website by using the crawler robot, and storing the current latest data into a database;
the database is used for storing the current latest data corresponding to each preset user;
and the listener module is used for monitoring the current latest data stored in the database, acquiring the latest data updated by the preset user and sending the latest data to a preset live broadcast platform.
Optionally, the access authorization module includes:
and the token acquisition unit is used for simulating a user to acquire a token which is corresponding to the authorization information and permanently logs in the dynamic website through an Oauth2.0 protocol according to the authorization information.
Optionally, the access authorization module includes:
a sending unit, configured to send an authorization request for obtaining the authorization information to the selected user;
and the receiving unit is used for acquiring the authorization information returned by the selected user.
Optionally, the data obtaining module includes:
the allocation unit is used for allocating a preset number of the crawler robots to each preset user;
and the acquisition unit is used for acquiring the current latest data by the crawler robots corresponding to the preset users in sequence within respective preset time.
Optionally, the system further comprises:
and the counting module is used for counting all the latest data corresponding to each preset user through a fixed platform website according to preset indexes and displaying a counting result.
The live broadcast method of the dynamic website provided by the invention comprises the following steps: obtaining a token which is corresponding to the authorization information and is permanently logged in the dynamic website according to the authorization information of a selected user in the dynamic website; the authorization information comprises login information of a corresponding selected user for logging in the dynamic website; allocating a crawler robot to each token, acquiring current latest data of a preset user of the dynamic website by using the crawler robot, and storing the current latest data into a database; and monitoring the current latest data stored in the database, acquiring the latest data updated by the preset user, and sending the latest data to a preset live broadcast platform.
According to the method and the system, the token which is corresponding to the authorization information and permanently logs in the dynamic website is obtained according to the authorization information of the selected user in the dynamic website, so that the crawler robot can permanently log in the dynamic website through the token, the access limit of the dynamic website to the crawler robot is solved, the crawler robot is allocated to each token, the current latest data of the preset user of the dynamic website is obtained through the crawler robot, the current latest data of the preset user of the dynamic website can be captured through a plurality of crawler robots, the limitation of the dynamic website such as the Xinlang microblog on the crawling frequency of the crawler robot is effectively avoided, the live broadcast real-time performance is improved, and the user experience is improved. In addition, the invention also provides a live broadcast system of the dynamic website, and the live broadcast system also has the beneficial effects.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.
Fig. 1 is a flowchart of a live broadcast method for a dynamic website according to an embodiment of the present invention;
fig. 2 is a schematic system structure diagram of a live broadcast method for a dynamic website according to an embodiment of the present invention.
Fig. 3 is a flowchart of another live broadcasting method for a dynamic website according to an embodiment of the present invention;
fig. 4 is a schematic diagram illustrating token acquisition in another live broadcast method for a dynamic website according to an embodiment of the present invention;
fig. 5 is a schematic system flow diagram of another live broadcasting method for a dynamic website according to an embodiment of the present invention;
fig. 6 is a structural diagram of a live broadcast system of a dynamic website according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are some, but not all, embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
Referring to fig. 1 and fig. 2, fig. 1 is a flowchart illustrating a live broadcast method for a dynamic website according to an embodiment of the present invention; fig. 2 is a schematic system structure diagram of a live broadcast method for a dynamic website according to an embodiment of the present invention. The method can comprise the following steps:
step 101: obtaining a token which is corresponding to the authorization information and is permanently logged in the dynamic website according to the authorization information of a selected user in the dynamic website; and the authorization information comprises login information of a corresponding selected user for logging in the dynamic website.
It will be appreciated that the selected user may be an existing user in the dynamic website, i.e., a registered user in the dynamic website. The specific obtaining mode of the authorization information of the selected user can be that an authorization request for obtaining the authorization information is sent to the selected user, the authorization information returned by the selected user is received, namely the authorization request for obtaining the authorization information is sent to any registered user in the dynamic website, the authorization information returned by the registered user is received, wherein the registered user of the returned authorization information is taken as the selected user; the authorization information of the selected user can also be directly obtained, that is, the user can be registered in the dynamic website specially, and the authorization information of the registered user can be directly obtained, wherein the registered user can be the selected user. As long as the corresponding token can be obtained according to the authorization information of the selected user, the specific obtaining mode of the authorization information of the selected user and the specific personnel selection of the selected user can be set by the designer according to the practical scenario and the user requirement, and this embodiment does not limit this.
It should be noted that, as to the specific manner of obtaining the token of the dynamic website permanently logged in corresponding to the authorization information through the authorization information in this step, the oauth2.0 technology may be used, that is, according to the authorization information, the user is simulated to obtain the token corresponding to the authorization information through the oauth2.0 protocol; other techniques may also be used. As long as the token for permanently logging in the dynamic website can be obtained through the authorization information, the specific way of obtaining the token is not limited in this embodiment.
Specifically, in this step, as shown in fig. 2, the access authorization module of the live broadcast system of the dynamic website sends authorization information to the dynamic website, and receives a token (access robot) corresponding to the authorization information returned by the dynamic website. The present embodiment is not limited to this.
Step 102: and allocating a crawler robot to each token, acquiring the current latest data of a preset user of the dynamic website by using the crawler robot, and storing the current latest data into a database.
In the step, by using a specific mode that the crawler robot acquires the current latest data of the preset user of the dynamic website, each crawler robot can log in the dynamic website by acquiring an IP address from an IP proxy server through a token, and then find the preset user and acquire the current latest data of the preset user within the preset safe crawling time; the present embodiment does not limit the specific obtaining manner, as long as the crawler robot can be used to obtain the current latest data of the preset user of the dynamic website.
It can be understood that, because a dynamic website such as a Sing microblog may refuse to access an IP of the dynamic website for a certain period of time if a request from the IP is too frequent in a certain period of time, that is, the crawling frequency of each crawler robot is limited, a manner that a plurality of crawler robots collectively obtain current latest data of preset users of the dynamic website through this step may be used, for example, a preset number of crawler robots are allocated to each preset user, and the crawler robot corresponding to each preset user sequentially obtains current latest data within their respective preset time. The present embodiment does not set any limit to this.
It should be noted that, in this step, the preset user may be a user needing live broadcast in the dynamic website, for example, the preset user may be a stock market large V needing live broadcast in the green microblog. The current latest data of the preset user can be the latest data released by the preset user, and for example, the current latest data can be the latest microblog released by the big-tv stock market in the green-wave microblog which needs to be live broadcasted. The specific content of the preset user and the current latest data can be set by the designer according to the practical scene and the user requirement, and this embodiment does not limit this.
Specifically, in this step, as shown in fig. 2, an access authorization module of the live broadcast system of the dynamic website sends a token (access token) to a data acquisition module, the data acquisition module allocates a crawler Robot (crawler Robot) in a crawler Robot group to each token, each crawler Robot acquires different respective IPs through an IP proxy server, and the crawling is performed on current latest data of a preset user in the dynamic website within a safe crawling time.
Step 103: and monitoring the current latest data stored in the database, acquiring the latest data updated by the preset user, and sending the latest data to a preset live broadcast platform.
The updated latest data of the preset user can be the current latest data updated by the preset user and acquired by the first crawler robot and stored in the database, for example, when the plurality of crawler robots acquire the latest microblog issued by a large V stock market in the green microblog, after the large V stock market updates one microblog, that is, after one microblog is issued, the updated latest data of the preset user can be the microblog acquired by the first crawler robot and stored in the database.
It can be understood that, as long as it is ensured that the updated latest data of the preset user can be obtained, as for a specific manner of monitoring the current latest data stored in the database in this step, as shown in fig. 2, the listener module of the live broadcast system of the dynamic website may continuously poll the database through a data interface (Restful) provided by WebService until the updated latest data of the preset user is found, and other manners may also be used, which is not limited in this embodiment.
It should be noted that the preset live broadcast platform may be a QQ group and/or a wechat group and/or a platform live broadcast website. Specifically, in this step, as shown in fig. 2, the listener module of the dynamic website live broadcast system continuously polls the database through a data interface (Restful) provided by WebService until the latest data updated by the preset user is found, and pushes the latest data to the QQ group, the wechat group, and the platform live broadcast website for live broadcast.
Preferably, the method provided by this embodiment may further include: and counting all the latest data corresponding to each preset user according to preset indexes through a fixed platform website and displaying a counting result. That is to say, with the method provided in this embodiment, not only the latest data updated by the preset user can be live broadcast through the QQ group, the wechat group, and the platform live broadcast website, but also the related statistics can be performed on the past live broadcast data through the platform website, so as to improve the user experience.
In the embodiment of the invention, the token which is corresponding to the authorization information and permanently logs in the dynamic website is obtained according to the authorization information of the selected user in the dynamic website, so that the crawler robot can permanently log in the dynamic website through the token, the access limit of the dynamic website to the crawler robot is solved, the crawler robot is allocated to each token, the current latest data of the preset user of the dynamic website is obtained through the crawler robot, the current latest data of the preset user of the dynamic website can be captured through a plurality of crawler robots, the limitation of the dynamic website such as the Xinlang microblog on the crawling frequency of the crawler robot is effectively avoided, the live broadcast real-time performance is improved, and the user experience is improved.
Please refer to fig. 3, fig. 4, fig. 5, and fig. 3, which are flowcharts illustrating another live broadcasting method for a dynamic website according to an embodiment of the present invention; fig. 4 is a schematic diagram illustrating token acquisition in another live broadcast method for a dynamic website according to an embodiment of the present invention; fig. 5 is a system flow diagram illustrating another live broadcasting method for a dynamic website according to an embodiment of the present invention. The method can comprise the following steps:
step 201: and sending an authorization request for obtaining authorization information to the selected user.
In this step, an authorization request may be sent to all or part of registered users in the dynamic website, that is, as shown in fig. 4, the dynamic website live broadcast system serves as a resource requester to request authorization from a resource owner (registered user) through an interface provided by the dynamic website.
It can be understood that, the selection manner of the part of the registered users for sending the authorization request may be set by the designer according to the practical scenario, and this embodiment does not limit this.
Step 202: and acquiring the authorization information returned by the selected user.
Wherein, the selected user can be a registered user returning the authorization information. This step may be illustrated in fig. 4, where the resource owner gives authorization to the resource requestor.
It should be noted that, as long as the token for permanently logging in the dynamic website can be obtained through the authorization information, the embodiment does not make any limitation on obtaining the specific content of the authorization information returned by the selected user, that is, the specific content included in the authorization information.
Step 203: and obtaining a token which is corresponding to the authorization information and is permanently logged in the dynamic website according to the authorization information of the selected user in the dynamic website.
As shown in fig. 4, in this step, the live broadcast system of the dynamic website, as a resource requester, first sends authorization information to the authorization server of the dynamic website to request to acquire an access token, and then receives the token issued by the authorization server after the authorization server verifies the authorization information.
Step 204: one crawler robot is assigned to each token.
It will be appreciated that each crawler robot has a different token.
Step 205: and allocating a preset number of crawler robots to each preset user.
The preset number in the step can be set by a designer according to a practical scene and user requirements, if the crawler robots are required to crawl microblog data in the green wave microblog, 20 crawler robots can be allocated to each large V account needing to crawl the microblog, so that the latest microblog of the large V account can be crawled in real time on the basis of avoiding the limitation of the green wave microblog on the crawling frequency of a single crawler robot, and the delay of live broadcast of the large V account is reduced. The present embodiment is not limited to this.
Step 206: and the crawler robots corresponding to each preset user sequentially acquire current latest data within respective preset time, and store the current latest data into a database.
As can be understood, for the way of acquiring the current latest data of each preset user by the preset number of crawler robots corresponding to each preset user, the preset number of crawler robots can sequentially acquire the current latest data of the preset user within the respective safe crawling time according to the preset sequence; other obtaining modes can also be used, for example, a preset number of crawler robots corresponding to each preset user sequentially obtain the current latest data of the preset user according to a preset time interval. As long as the predetermined number of crawler robots obtain the current latest data of the predetermined user on the basis that the access of the predetermined number of crawler robots by the dynamic website is not limited, the present embodiment does not make any limitation on the manner of obtaining the current latest data of the predetermined user by the predetermined number of crawler robots corresponding to each predetermined user.
It should be noted that, in this step, as shown in fig. 4, the live broadcasting system of the dynamic website serves as a resource requester, and the crawler robot with the access token acquires the target resource, that is, the current latest data of the preset user, from the resource server of the dynamic website.
Specifically, the data corresponding to the preset user acquired by the crawler robot may be current latest data, such as a current latest microblog of a large V account number of a microblog to be crawled in a green microblog; the updated latest data of the user can also be preset, for example, a preset number of crawler robots only obtain the updated latest microblogs of the large V account needing to crawl the microblogs in one newly-surfed microblog, that is, after only one crawler robot obtains and stores the latest microblogs, other crawler robots do not obtain and store the latest microblogs any more. The present embodiment does not set any limit to this.
Step 207: monitoring current latest data stored in the database, acquiring the latest data updated by a preset user, and sending the latest data to a preset live broadcast platform.
In this step, the current latest data stored in the database is monitored, and the latest data updated by the preset user is obtained, as shown in fig. 5, the monitor module of the live broadcast system of the dynamic website continuously polls the database through the data interface provided by WebService, queries whether the current latest data stored in the database has the latest data updated by the preset user, and if so, returns the latest data. The Restful technology can be adopted for the design of the data interface, and each time the application operates the database, the operation can be regarded as one request for data service, so that each type of data operation can be mapped into a corresponding data request mode in an http protocol, and the data operation can be realized by utilizing a simple and clear url. The present embodiment does not set any limit to this.
It can be understood that, for the specific mode of sending the latest data to the preset live broadcast platform, the setting can be correspondingly set by designers according to the type of the preset live broadcast platform, and if the preset live broadcast platform is a pre-established QQ group and a wechat group, the keyboard operation can be simulated by using a script program, and the latest data is automatically issued to the group, so as to realize live broadcast. The present embodiment does not set any limit to this.
Step 208: and counting all the latest data corresponding to each preset user according to preset indexes through a fixed platform website and displaying a counting result.
In this step, the latest data of all the preset users may be counted and the statistical result may be displayed, or the latest data of one or more of all the preset users may be counted and the statistical result may be displayed, which is not limited in this embodiment.
It can be understood that, for a specific way of counting all the latest data corresponding to each preset user according to the preset index, the method can be set by an actual person according to a practical scene and user requirements, for example, the bin information in all live microblogs of stock market large V in a micro-wave microblog is counted, a curve graph of bin change is obtained, and the curve graph is displayed on a fixed platform website.
In the embodiment, a preset number of crawler robots are allocated to each preset user, the crawler robots corresponding to each preset user sequentially acquire current latest data within respective preset time, and the current latest data are stored in the database, so that the limitation of dynamic websites such as the Xinlang microblog on the crawling frequency of the crawler robots can be effectively avoided, and the real-time performance of live broadcast is improved; all latest data corresponding to each preset user are counted and statistical results are displayed through the fixed platform website according to preset indexes, related statistics can be carried out on the past live broadcast data through the platform website, and user experience is improved.
Referring to fig. 6, fig. 6 is a structural diagram of a live broadcast system of a dynamic website according to an embodiment of the present invention. The system may include:
the access authorization module 100 is configured to obtain a token, corresponding to authorization information, for permanently logging in a dynamic website according to the authorization information of a selected user in the dynamic website; the authorization information comprises login information of a corresponding selected user for logging in the dynamic website;
the data acquisition module 200 is configured to allocate a crawler robot to each token, acquire current latest data of a preset user of the dynamic website by using the crawler robot, and store the current latest data in the database 300;
the database 300 is configured to store the current latest data corresponding to each preset user;
the listener module 400 is configured to monitor the current latest data stored in the database 300, obtain the latest data updated by the preset user, and send the latest data to a preset live broadcast platform.
Optionally, the access authorization module 100 may include:
and the token acquisition unit is used for simulating a user to acquire a token which is corresponding to the authorization information and permanently logs in the dynamic website through an Oauth2.0 protocol according to the authorization information.
Optionally, the access authorization module 100 may include:
a sending unit, configured to send an authorization request for obtaining the authorization information to the selected user;
and the receiving unit is used for acquiring the authorization information returned by the selected user.
Optionally, the data obtaining module 200 may include:
the allocation unit is used for allocating a preset number of the crawler robots to each preset user;
and the acquisition unit is used for acquiring the current latest data by the crawler robots corresponding to the preset users in sequence within respective preset time.
Optionally, the system may further include:
and the counting module is used for counting all the latest data corresponding to each preset user through a fixed platform website according to preset indexes and displaying a counting result.
In the embodiment of the invention, the access authorization module 100 acquires the token permanently logging in the dynamic website corresponding to the authorization information according to the authorization information of the selected user in the dynamic website, so that the crawler robot can permanently log in the dynamic website through the token, the access limitation of the dynamic website to the crawler robot is solved, one crawler robot is allocated to each token through the data acquisition module 200, the current latest data of the preset user of the dynamic website is acquired through the crawler robot, the current latest data of the preset user of the dynamic website can be captured through a plurality of crawler robots, the limitation of the dynamic website such as a Xinlang microblog on the crawling frequency of the crawler robot is effectively avoided, the live broadcast real-time performance is improved, and the user experience is improved.
The embodiments are described in a progressive manner in the specification, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device disclosed by the embodiment corresponds to the method disclosed by the embodiment, so that the description is simple, and the relevant points can be referred to the method part for description.
Those of skill would further appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate this interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The live broadcasting method and system of the dynamic website provided by the invention are introduced in detail above. The principles and embodiments of the present invention are explained herein using specific examples, which are presented only to assist in understanding the method and its core concepts. It should be noted that, for those skilled in the art, it is possible to make various improvements and modifications to the present invention without departing from the principle of the present invention, and those improvements and modifications also fall within the scope of the claims of the present invention.
Claims (6)
1. A live broadcast method of a dynamic website is characterized by comprising the following steps:
obtaining a token which is corresponding to the authorization information and is permanently logged in the dynamic website according to the authorization information of a selected user in the dynamic website; the authorization information comprises login information of a corresponding selected user for logging in the dynamic website;
allocating a crawler robot to each token, acquiring current latest data of a preset user of the dynamic website by using the crawler robot, and storing the current latest data into a database;
monitoring the current latest data stored in the database, acquiring the latest data updated by the preset user, and sending the latest data to a preset live broadcast platform;
the method for acquiring the current latest data of the preset user of the dynamic website by using the crawler robot comprises the following steps:
distributing a preset number of the crawler robots to each preset user;
the crawler robot corresponding to each preset user sequentially acquires the current latest data within respective preset time;
and counting all the latest data corresponding to each preset user according to preset indexes through a fixed platform website and displaying a counting result.
2. The live broadcasting method of a dynamic website according to claim 1, wherein the obtaining a token permanently logging in the dynamic website corresponding to the authorization information according to the authorization information of the selected user in the dynamic website comprises:
and according to the authorization information, simulating a user to obtain a token which is corresponding to the authorization information and permanently logs in the dynamic website through an Oauth2.0 protocol.
3. The live broadcasting method of a dynamic website as claimed in claim 2, wherein before simulating, according to the authorization information, that the user obtains the token permanently logging in the dynamic website corresponding to the authorization information via the oauth2.0 protocol, the method further comprises:
sending an authorization request for obtaining the authorization information to the selected user;
and acquiring the authorization information returned by the selected user.
4. A live broadcast system for a dynamic website, comprising:
the access authorization module is used for acquiring a token which is corresponding to the authorization information and is used for permanently logging in the dynamic website according to the authorization information of the selected user in the dynamic website; the authorization information comprises login information of a corresponding selected user for logging in the dynamic website;
the data acquisition module is used for allocating a crawler robot to each token, acquiring the current latest data of a preset user of the dynamic website by using the crawler robot, and storing the current latest data into a database;
the database is used for storing the current latest data corresponding to each preset user;
the listener module is used for monitoring the current latest data stored in the database, acquiring the latest data updated by the preset user and sending the latest data to a preset live broadcast platform;
the statistical module is used for performing statistics on all the latest data corresponding to each preset user through a fixed platform website according to preset indexes and displaying statistical results;
wherein, the data acquisition module includes:
the allocation unit is used for allocating a preset number of the crawler robots to each preset user;
and the acquisition unit is used for acquiring the current latest data by the crawler robots corresponding to the preset users in sequence within respective preset time.
5. The live broadcast system of the dynamic website as claimed in claim 4, wherein the access authorization module comprises:
and the token acquisition unit is used for simulating a user to acquire a token which is corresponding to the authorization information and permanently logs in the dynamic website through an Oauth2.0 protocol according to the authorization information.
6. The live broadcast system of the dynamic website as claimed in claim 5, wherein the access authorization module comprises:
a sending unit, configured to send an authorization request for obtaining the authorization information to the selected user;
and the receiving unit is used for acquiring the authorization information returned by the selected user.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710278347.6A CN107103079B (en) | 2017-04-25 | 2017-04-25 | Live broadcast method and system for dynamic website |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710278347.6A CN107103079B (en) | 2017-04-25 | 2017-04-25 | Live broadcast method and system for dynamic website |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN107103079A CN107103079A (en) | 2017-08-29 |
| CN107103079B true CN107103079B (en) | 2021-05-25 |
Family
ID=59657310
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710278347.6A Active CN107103079B (en) | 2017-04-25 | 2017-04-25 | Live broadcast method and system for dynamic website |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107103079B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110620670A (en) * | 2019-10-15 | 2019-12-27 | 深圳市小赢信息技术有限责任公司 | Token acquisition method, data acquisition system, proxy server, and storage medium |
Family Cites Families (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7996349B2 (en) * | 2007-12-05 | 2011-08-09 | Yahoo! Inc. | Methods and apparatus for computing graph similarity via sequence similarity |
| CN101551813A (en) * | 2009-05-13 | 2009-10-07 | 腾讯科技（深圳）有限公司 | Network connection apparatus, search equipment and method for collecting search engine data source |
| CN103037010A (en) * | 2012-12-26 | 2013-04-10 | 人民搜索网络股份公司 | Distributed network crawler system and catching method thereof |
| CN103440139A (en) * | 2013-09-11 | 2013-12-11 | 北京邮电大学 | Acquisition method and tool facing microblog IDs (identitiesy) of mainstream microblog websites |
-
2017
- 2017-04-25 CN CN201710278347.6A patent/CN107103079B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN107103079A (en) | 2017-08-29 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110147398B (en) | Data processing method, device, medium and electronic equipment | |
| CN107395683B (en) | Method for selecting return path and server | |
| US9769248B1 (en) | Performance-based content delivery | |
| US10027739B1 (en) | Performance-based content delivery | |
| Williams et al. | Web workload characterization: Ten years later | |
| CN108737467B (en) | Server log viewing method, device and system | |
| CN105024872B (en) | The method and device of applied in network performance test | |
| EP3170091B1 (en) | Method and server of remote information query | |
| CN110324680B (en) | Video pushing method and device, server, client and storage medium | |
| CN103685590B (en) | Obtain the method and system of IP address | |
| CN102624920A (en) | Method and device for performing access through proxy server | |
| CN110609937A (en) | Crawler identification method and device | |
| CN111787345B (en) | Interactive resource processing method and device based on network live broadcast room, server and storage medium | |
| CN104219230B (en) | Identify method and the device of malicious websites | |
| CN107347015B (en) | Method, device and system for identifying content distribution network | |
| CN109063158B (en) | Method, device, system and medium for inquiring website access ranking information | |
| CN108429777A (en) | Data updating method based on cache and server | |
| CN111753223A (en) | Access control method and device | |
| US20170141994A1 (en) | Anti-leech method and system | |
| CN107147662B (en) | Domain name hijacking discovery method | |
| CN106713456B (en) | Network bandwidth statistical method and device | |
| CN111966967A (en) | Copyright storage method and system based on block chain technology and CDN | |
| CN113542418B (en) | File management method, device, electronic equipment and storage medium | |
| Li et al. | Challenges, designs, and performances of large-scale open-P2SP content distribution | |
| CN107103079B (en) | Live broadcast method and system for dynamic website |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| CB02 | Change of applicant information |
Address after: 215347 7th floor, IIR complex, 1699 Weicheng South Road, Kunshan City, Suzhou City, Jiangsu Province Applicant after: Kunshan Microelectronics Technology Research Institute Address before: 215347 7th floor, complex building, No. 1699, Zuchongzhi South Road, Kunshan City, Suzhou City, Jiangsu Province Applicant before: KUNSHAN BRANCH, INSTITUTE OF MICROELECTRONICS OF CHINESE ACADEMY OF SCIENCES |
|
| CB02 | Change of applicant information | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |