CN103186613A - Movie and television resource aggregation system - Google Patents
Movie and television resource aggregation system Download PDFInfo
- Publication number
- CN103186613A CN103186613A CN 201110456861 CN201110456861A CN103186613A CN 103186613 A CN103186613 A CN 103186613A CN 201110456861 CN201110456861 CN 201110456861 CN 201110456861 A CN201110456861 A CN 201110456861A CN 103186613 A CN103186613 A CN 103186613A
- Authority
- CN
- China
- Prior art keywords
- task
- data
- resource
- server
- url
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention discloses a movie and television resource aggregation system. The movie and television resource aggregation system is characterized by comprising the following steps: starting a crawler; initializing parameters and starting a multi-thread background task; scanning the task, requesting a data interface, acquiring a task list and automatically generating by a rule or manually adding; judging whether the task exists or not, requesting to monopolize the task, requesting a uniform resource locator (URL) and analyzing data; and finally compressing data, uploading the data to a data server and finishing data processing. Resource aggregation can aggregate the contents of various websites to our own server, so that users can see the contents of various websites at one client side.
Description
Technical field
The present invention relates to a kind of video display resource paradigmatic system.
Background technology
At present general general way is to do a link in the website, and oneself does not preserve detailed information.Existing common practice can't accomplish to allow the user experience unanimity, can not accomplish comprehensive search.Sometimes in order to find best Internet resources can use a plurality of search engines to search for and compare simultaneously, but it is more loaded down with trivial details to operate.And changes in network resources is very fast, if do not remember the Search Results link at that time or in time preserved webpage, just may can not find next time.The user can find various film resources in a place for convenience, does not need each website to search.Be necessary to set up a resource paradigmatic system.
Summary of the invention
The present invention is directed to the proposition of above problem, and develop a kind of video display resource paradigmatic system.The technological means that the present invention adopts is as follows:
A kind of video display resource paradigmatic system is characterized in that comprising the steps:
At first, start reptile;
Then, initiation parameter starts the multithreading background task; Scan task, the request msg interface obtains task list, is generated automatically or manually interpolation by rule; Whether task is arranged, and task, request URL, resolution data are monopolized in request;
At last, packed data also uploads to data server, finishes data and handles.
The rear end of resource polymerization is a distributed web service system based on Linux, adopts the scheme of Nginx+Tomcat+Memcache.
Nginx is a high-performance Web and Reverse Proxy, connects under the concurrent situation at height; Backend application API server uses Tomcat as the container of Service Component, and its principle of design is followed RESTful software architecture thought, comes the structure of design system by url.According to REST, each url represents a resource, and total system is made up of these resource.
The resource polymerization can focus on the content of each home Web site our server, and the user can see the content of each home Web site a client.
Description of drawings
Fig. 1 is workflow diagram of the present invention;
Fig. 2 is structured flowchart of the present invention.
Embodiment
Video display resource paradigmatic system as depicted in figs. 1 and 2 comprises the steps: at first, starts reptile; Then, initiation parameter starts the multithreading background task; Scan task, the request msg interface obtains task list, is generated automatically or manually interpolation by rule; Whether task is arranged, and task, request URL, resolution data are monopolized in request; At last, packed data also uploads to data server, finishes data and handles.
The collection of resources program is gathered webpage from each website that needs the resource polymerization, if necessary, can recurrence gather the deeper webpage that the link on the webpage is pointed to, analyze and obtain wherein film details and broadcast address, submitting to the background application server puts in order, classification also deposits database in, and like this, lane database has just been preserved the film information of each website; The user selects to browse or use search by application when using client to browse; This polymerization methods is most important for user's experience, and the user can be faster by the resource polymerization, finds the thing of oneself wanting more accurately; The resource aggregated service is that typical client-server is used, and background system provides interface and service by tomcat, and it is between client and database.
The rear end of whole resource polymerization is a distributed web service system based on Linux, adopts the scheme of Nginx+Tomcat+Memcache, is easy to expand and large scale deployment.Nginx is a high-performance Web and Reverse Proxy, connect under the concurrent situation at height, Nginx is the pretty good substitute of Apache Server: Nginx has become the fictitious host computer operation in the whole world first-selected software platform, can support up to 50, the response of 000 concurrent connection number, committed memory is few, and Nginx can also use as load-balanced server simultaneously.Backend application API server uses Tomcat as the container of Service Component, and its principle of design is followed RESTful software architecture thought, comes the structure of design system by url.According to REST, each url represents a resource, and total system is made up of these resource.REST is not only the data transport agreement of HTTP as oneself, and the instrument that data are handled is directly carried out in conduct.Why REST can improve the scalability of system, is because its forces all operations all is stateless, does not so just have the constraint of context, if do distributed, do cluster, considered the problem of context with regard to not needing.Simultaneously, it makes system can use pool effectively.Another of the performance of REST promotes the service of only being responsible for providing resource and operation resource from its distribution to clint and server task: server, and client will be according to the data among the resource and representation do-it-yourself render.This has just reduced server expenses.Service end makes client needn't know the realization details of remote service based on the design of interface.Do not make any modification in client code, implementation just can change.Adopt the API of REST style, client call is also extremely convenient.
The language and the framework that adopt: system adopts Java development language and Spring J2EE Development Framework to realize remote service cross-platform, Noninvasive, lightweight, it uses technology such as Aspects and AOP, oppositely control (IoC), Design Modes such as agency and factory mode, the realization details of business task in can the packing service class is only left interface for client.
The exchanges data lattice JSON (JavaScript Object Notation) of lightweight is adopted in communication between service end and the client, is easy to the people and reads and write.Also being easy to simultaneously machine resolves and generates.By our high speed transmission technology, even under the concurrent situation of height, also can be controlled in a rational acceptable degree to taking of the network bandwidth again.The all data of caching technology Content Management System that system adopts all are kept in the middle of the database basically, access database frequently causes database performance sharply to descend at every turn, can't serve more user simultaneously, such as MySQL, may cause locking especially frequently table.Given this Content Management System uses Memcache as centralized caching system, allow share the pressure of database, Memcache is the project of danga.com, and at present a lot of systems in the whole world use this cache item to make up the website of own heavy load, share the pressure of database.It can tackle any a plurality of connection, uses the network I/O of unblock.Because data are stored in the distributed internal memory, the speed of data access is very fast.
The database technology that system adopts: Content Management System uses MySQL as the Back end data library storage system.MySQL is most popular, open source code, high-speed, high-performance, multithreading, is based upon the structural Relational DBMS of client/server (Client/Server) (RDBMS).The developer of MySQL pays close attention to its performance from the beginning always, stint no sacrifice feature set for this reason, and up to today, MySQL still keeps true qualities, and at full speed high-performance is First Principles.As time goes on, MySQL has also added the advanced feature of large database product, as storing process, view, trigger, affairs etc., makes it begin to be deployed application in the enterprise-level database system.
The load-balancing technique that system adopts: architectural framework and the deployment of the active and standby MySQL of the many Tomcat+ of many Nginx+ adopted in Web service, can realize the load balancing of cluster easily, and specific polling flexibly, based on visit IP's or based on the route distribution strategy of weight.If rear end Tomcat server down falls, can also reject automatically.Native system is supported to share SESSION between many TOMCAT servers, guarantees the validity of cluster on using.To be a main MySQL server (Master) server with a plurality of subordinate MySQL servers (Slave) set up back-end data base copies (replication) and be connected, main DB server and subordinate DB server are realized data sync to a certain extent, the identical data trnascription of a plurality of dependent server storages, realize data redundancy, fault tolerance is provided.When disposing the Application and Development system, the database operation code is optimized, and (as UPDATE, INSERT) is directed to master server with write operation, and (SELECT) is directed to dependent server a large amount of query manipulations, realize the load-balancing function of cluster, thereby obtain better customer response time.
The above; only be the preferable embodiment of the present invention; but protection scope of the present invention is not limited thereto; anyly be familiar with those skilled in the art in the technical scope that the present invention discloses; be equal to replacement or change according to technical scheme of the present invention and inventive concept thereof, all should be encompassed within protection scope of the present invention.
Claims (3)
1. a video display resource paradigmatic system is characterized in that comprising the steps:
At first, start reptile;
Then, initiation parameter starts the multithreading background task; Scan task, the request msg interface obtains task list, is generated automatically or manually interpolation by rule; Whether task is arranged, and task, request URL, resolution data are monopolized in request;
At last, packed data also uploads to data server, finishes data and handles.
2. a kind of video display resource paradigmatic system according to claim 1, the rear end that it is characterized in that the resource polymerization is a distributed web service system based on Linux, adopts the scheme of Nginx+Tomcat+Memcache.
3. a kind of video display resource paradigmatic system according to claim 1 is characterized in that Nginx is a high-performance Web and Reverse Proxy, connects under the concurrent situation at height; Backend application API server uses Tomcat as the container of Service Component, its principle of design is followed RESTful software architecture thought, comes the structure of design system by url, according to REST, each url represents a resource, and total system is made up of these resource.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110456861 CN103186613A (en) | 2011-12-30 | 2011-12-30 | Movie and television resource aggregation system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110456861 CN103186613A (en) | 2011-12-30 | 2011-12-30 | Movie and television resource aggregation system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103186613A true CN103186613A (en) | 2013-07-03 |
Family
ID=48677782
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110456861 Pending CN103186613A (en) | 2011-12-30 | 2011-12-30 | Movie and television resource aggregation system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103186613A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103605527A (en) * | 2013-12-06 | 2014-02-26 | 普元信息技术股份有限公司 | System and method for realizing cloud computing application interface integration and unified style display |
| CN104462431A (en) * | 2014-12-16 | 2015-03-25 | 浪潮软件集团有限公司 | Method for crawling web page recruitment information |
| CN104598840A (en) * | 2015-02-04 | 2015-05-06 | 新余兴邦信息产业有限公司 | Multi-data-source data processing method and system based on HTTP (hyper text transport protocol) |
| CN106126693A (en) * | 2016-06-29 | 2016-11-16 | 微梦创科网络科技（中国）有限公司 | The sending method of the related data of a kind of webpage and device |
-
2011
- 2011-12-30 CN CN 201110456861 patent/CN103186613A/en active Pending
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103605527A (en) * | 2013-12-06 | 2014-02-26 | 普元信息技术股份有限公司 | System and method for realizing cloud computing application interface integration and unified style display |
| CN104462431A (en) * | 2014-12-16 | 2015-03-25 | 浪潮软件集团有限公司 | Method for crawling web page recruitment information |
| CN104598840A (en) * | 2015-02-04 | 2015-05-06 | 新余兴邦信息产业有限公司 | Multi-data-source data processing method and system based on HTTP (hyper text transport protocol) |
| WO2016123921A1 (en) * | 2015-02-04 | 2016-08-11 | 新余兴邦信息产业有限公司 | Http protocol-based multiple data resource data processing method and system |
| CN106126693A (en) * | 2016-06-29 | 2016-11-16 | 微梦创科网络科技（中国）有限公司 | The sending method of the related data of a kind of webpage and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11029940B2 (en) | Maintaining client version affinity during a server cluster upgrade | |
| US11240289B2 (en) | Apparatus and method for low-latency message request/response processing | |
| US20200065295A1 (en) | Method for connecting a relational data store's meta data with hadoop | |
| JP7093601B2 (en) | Methods, devices, and systems for group-based communication systems that interact with remote resources for remote data objects. | |
| JP6334009B2 (en) | Handling mutations for remote databases | |
| US9268716B2 (en) | Writing data from hadoop to off grid storage | |
| EP3327587B1 (en) | Workload shifting in a database system using hint-based routing | |
| JP5719323B2 (en) | Distributed processing system, dispatcher and distributed processing management device | |
| US9576067B2 (en) | Enhancing client-side object caching for web based applications | |
| US9460151B2 (en) | System and methods to configure a query language using an operator dictionary | |
| US9110807B2 (en) | Cache conflict detection | |
| US9712622B2 (en) | Techniques to manage state information for a web service | |
| US20160103914A1 (en) | Offloading search processing against analytic data stores | |
| US20210352074A1 (en) | Method, apparatus, and computer program product for managing access permissions for a searchable enterprise platform | |
| EP3035216A1 (en) | Cloud bursting a database | |
| CN102082800A (en) | User request processing method and server | |
| US20170344539A1 (en) | System and method for improved scalability of database exports | |
| US20160028833A1 (en) | Tenant aware session manager | |
| US11005957B2 (en) | Distributed generation of live feed updates in a social networking system | |
| WO2019118867A1 (en) | Method, apparatus and computer program product for improving data indexing in a group-based communication platform | |
| CN103186613A (en) | Movie and television resource aggregation system | |
| US10827035B2 (en) | Data uniqued by canonical URL for rest application | |
| US8874682B2 (en) | Composite graph cache management | |
| US9401953B2 (en) | Intelligent high-volume cloud application programming interface request caching | |
| Zhygmanovskyi et al. | Cloud service provisioning based on peer-to-peer network for flexible service sharing and discovery |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C05 | Deemed withdrawal (patent law before 1993) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20130703 |