CN102169505A - Recommendation system building method based on cloud computing - Google Patents
Recommendation system building method based on cloud computing Download PDFInfo
- Publication number
- CN102169505A CN102169505A CN 201110125663 CN201110125663A CN102169505A CN 102169505 A CN102169505 A CN 102169505A CN 201110125663 CN201110125663 CN 201110125663 CN 201110125663 A CN201110125663 A CN 201110125663A CN 102169505 A CN102169505 A CN 102169505A
- Authority
- CN
- China
- Prior art keywords
- algorithm
- distributed
- mahout
- make
- cloud computing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a recommendation system building method based on cloud computing, which belongs to the field of cloud computing and recommendation system building. In the method, a Hadoop cloud platform with a plurality of nodes is built firstly; then a Mahout middleware is built on the Hadoop; a Mahout algorithm library is customized according to business demands; a traditional advancing algorithm, a pseudo-distributed advancing algorithm, and a distributed algorithm are realized on the Mahout middleware; and finally a recommended application framework is built according to the demands of users. With the invention, as a serial recommended algorithm and MapReduce are combined to realize a parallel algorithm, the processing efficiency can be effectively improved; a large quantity of data which can not be processed under the condition of using a single machine can be completed; and the recommendation results can be supplied to users very quickly.
Description
Technical field
The present invention relates to a kind of system constituting method, relate in particular to a kind of commending system construction method, belong to cloud computing and commending system and make up the field based on cloud computing.
Background technology
When providing increasing selection along with ecommerce, its structure also becomes and becomes increasingly complex, and the user can get lost in a large amount of merchandise news spaces, can't find the commodity that oneself need.A method that addresses this problem is a development intelligent recommendation system, be its Recommendations according to client's preference or demand, finishes purchasing process with the help client.Current have only a few large-scale e-commerce website (as Amazon, eBay etc.) at oneself business development personalized recommendation system, most medium and small sized enterprises do not have so much energy and resource and remove to drop into the commending system of building oneself.
The problem of recommending system to exist is at present: 1, underaction; 2, do not fully take into account business strategy; 3, can not adopt different recommendation strategies according to the variation of recommended requirements; 4, be difficult to handle large-scale data; 5, commending system need carry out customized development, integrated cost height, and transplantability is poor.These problems have limited commending system large-scale application in practice.
There are some challenges in concurrency based on the cloud computing data mining algorithm.Which type of algorithm handling present data mining with, this is a primary problem, and not all algorithm can both be finished present task with the mode of cloud computing.
Summary of the invention
The present invention is directed to the defective that existing commending system exists, and propose a kind of commending system construction method based on cloud computing.
This method comprises following content:
(1) make up the distributed file system layer:
Make up the Hadoop cloud platform of a plurality of nodes, one is host node in a plurality of nodes, and all the other are from node;
(2) make up the Distributed Calculation layer:
Adopt MapReduce as the distributed parallel computation model, on Hadoop, make up the Mahout middleware;
(3) customization data is analyzed middleware layer:
According to business demand customization Mahout algorithms library, realize that on the Mahout middleware tradition advances algorithm, pseudo-distributed propelling algorithm and distributed algorithm;
(4) make up the exemplary application layer:
According to the different demands of user, the correlation parameter size of algorithm in the Mahout algorithms library is set or calls algorithms of different, make up application framework;
According to the different demands of user, in application framework, call algorithms of different in the Mahout algorithms library, the correlation parameter size is set.
Technique effect:
The present invention proposes a commending system constructing plan general, that extendability is strong, this scheme combines project-based serial proposed algorithm and realizes parallel algorithm with MapReduce, formulated corresponding recommendation strategy flexibly, can effectively improve data-handling efficiency, can finish the mass data that to handle under the unit, and can fast recommendation results be fed back to the user.
Description of drawings
Fig. 1 is the commending system hierarchical chart.
Fig. 2 is the commending system Organization Chart.
Embodiment
Below the invention will be further described.
Ecommerce personalized recommendation system based on cloud computing comprises distributed file system layer, Distributed Calculation layer, data analysis middleware layer and exemplary application layer, wherein distributed file system layer and Distributed Calculation layer utilize Hadoop to make up, and the data analysis middleware layer is to customize on the basis of Mahout according to service application, and supplies user capture with method of service.Each layer of system all has corresponding cloud computing administration module, comprises data security, monitoring resource, resource dynamic scheduling, resource dynamic deployment, MRP and function such as virtual, guarantees that each layer all has high reliability and scalability.
The inventive method mainly comprises following content:
(1) make up the distributed file system layer:
Make up the Hadoop cloud platform of a plurality of nodes, one of them is a host node, and all the other are from node.This platform can provide distributed document storage and distributed programmed framework MapReduce.
(2) make up the Distributed Calculation layer:
On Hadoop, make up the Mahout middleware.Make up this middleware and can on this middleware, write distributed algorithm easily, carry out for Hadoop.
(3) customization data is analyzed middleware layer:
Realize that on Mahout tradition advances algorithm, pseudo-distributed propelling algorithm and distributed algorithm.These algorithms combine with distributed programmed framework MapReduce and write, and carry out so that distribute.
(4) make up the exemplary application layer:
Make up application framework, the exemplary application layer calls the algorithm of our definition according to the different demands of user, promptly the key parameter size in the algorithm is set or calls algorithms of different according to different demands.The data storage of user's input is on distributed file system, and exemplary application is executed on the Hadoop, executes the result and feeds back to the user.
Below the four systems layer is remarked additionally.
1) distributed file system layer
The present invention proposes to utilize Hadoop HDFS (distributed file system of Hadoop) to realize highly reliable distributed data file memory function, with the mass data distributed store on many computer clusters, file is carried out the piecemeal storage, duplicate for realizing the fault-tolerant piecemeal that carries out automatically.
2) Distributed Calculation layer
The present invention adopts MapReduce (parallel computational model) as the distributed parallel computation model, large-scale task is divided into a lot of fine-grained subtasks, these subtasks are distributed and dispatch and calculate on a plurality of computing nodes concurrently, thereby obtain the processing power to mass data on the cloud platform.
3) data analysis middleware layer
Mahout has realized data mining algorithms such as cluster, classification, collaborative filtering, evolution programming, and allows expansion, customizes our Mahout algorithms library according to the business demand of exemplary application layer, and calls for the exemplary application layer with method of service.The product process of typical electronic commercial affairs content recommendation comprises processes such as data modeling, data importing, data pre-service, recommended models processing, data output.The data analysis middleware layer relies on the Distributed Calculation layer and the distributed file system layer of lower floor, and the data set that can handle the mass data collection and increase fast is such as the user access logs that upgrades every day.Because Mahout depends on the most general cloud computing platform Hadoop, can coordinate to integrate with other cloud computing middleware effectively.
4) exemplary application layer
Realize ecommerce exemplary application such as information filtering recommendation, collaborative filtering recommending, mixing recommendation, satisfy the business demand of ecommerce.
Claims (3)
1. commending system construction method based on cloud computing is characterized in that:
This method comprises following content:
(1) make up the distributed file system layer:
Make up the Hadoop cloud platform of a plurality of nodes, one is host node in a plurality of nodes, and all the other are from node;
(2) make up the Distributed Calculation layer:
Adopt MapReduce as the distributed parallel computation model, on Hadoop, make up the Mahout middleware;
(3) customization data is analyzed middleware layer:
According to business demand customization Mahout algorithms library, realize that on the Mahout middleware tradition advances algorithm, pseudo-distributed propelling algorithm and distributed algorithm;
(4) make up the exemplary application layer:
According to the different demands of user, the correlation parameter size of algorithm in the Mahout algorithms library is set or calls algorithms of different, make up application framework;
According to the different demands of user, in application framework, call algorithms of different in the Mahout algorithms library, the correlation parameter size is set.
2. the commending system construction method based on cloud computing according to claim 1 is characterized in that: described tradition advances algorithm to combine with distributed programmed framework MapReduce and writes.
3. the commending system construction method based on cloud computing according to claim 1 is characterized in that: described exemplary application layer is executed on the Hadoop.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110125663 CN102169505A (en) | 2011-05-16 | 2011-05-16 | Recommendation system building method based on cloud computing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN 201110125663 CN102169505A (en) | 2011-05-16 | 2011-05-16 | Recommendation system building method based on cloud computing |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102169505A true CN102169505A (en) | 2011-08-31 |
Family
ID=44490667
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN 201110125663 Pending CN102169505A (en) | 2011-05-16 | 2011-05-16 | Recommendation system building method based on cloud computing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102169505A (en) |
Cited By (24)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102663019A (en) * | 2012-03-21 | 2012-09-12 | 北京英孚斯迈特信息技术有限公司 | Instant recommendation system |
| CN102724059A (en) * | 2012-03-31 | 2012-10-10 | 常熟市支塘镇新盛技术咨询服务有限公司 | Website operation state monitoring and abnormal detection based on MapReduce |
| CN103020282A (en) * | 2012-12-28 | 2013-04-03 | 深圳市彩讯科技有限公司 | General mixed association recommendation development platform and association recommendation method |
| CN103309867A (en) * | 2012-03-09 | 2013-09-18 | 句容智恒安全设备有限公司 | Web data mining system on basis of Hadoop platform |
| CN103345698A (en) * | 2013-07-09 | 2013-10-09 | 焦点科技股份有限公司 | Personalized recommendation method based on cloud processing mode and applied in e-business environment |
| CN103488775A (en) * | 2013-09-29 | 2014-01-01 | 中国科学院信息工程研究所 | Computing system and computing method for big data processing |
| CN103577403A (en) * | 2012-07-19 | 2014-02-12 | 镇江雅迅软件有限责任公司 | Cloud computing technology based recommendation system implementation method |
| CN103812679A (en) * | 2012-11-12 | 2014-05-21 | 深圳中兴网信科技有限公司 | Mass log statistical analysis system and method |
| CN104268709A (en) * | 2014-10-10 | 2015-01-07 | 浪潮集团有限公司 | Method for designing RFID system by distributed LSM tree |
| CN104503967A (en) * | 2014-10-24 | 2015-04-08 | 浪潮电子信息产业股份有限公司 | Hadoop-based network recommendation method |
| CN104834557A (en) * | 2015-05-18 | 2015-08-12 | 成都博元科技有限公司 | Data analysis method based on Hadoop |
| WO2016008317A1 (en) * | 2014-07-14 | 2016-01-21 | 华为技术有限公司 | Data processing method and central node |
| CN105321124A (en) * | 2015-11-23 | 2016-02-10 | 南京信息工程大学 | Hadoop-based electric power cloud platform design scheme |
| CN105677846A (en) * | 2016-01-06 | 2016-06-15 | 中国传媒大学 | Recommendation system and construction method thereof |
| CN106294439A (en) * | 2015-05-27 | 2017-01-04 | 北京广通神州网络技术有限公司 | A kind of data recommendation system and data recommendation method thereof |
| CN106339829A (en) * | 2016-11-10 | 2017-01-18 | 国网山东省电力公司济南供电公司 | Big data, Cloud, IoT and mobile internet technologies based active maintenance panorama monitoring system of power distribution network |
| CN106503140A (en) * | 2016-10-20 | 2017-03-15 | 安徽大学 | One kind is based on Hadoop cloud platform web resource personalized recommendation system and method |
| CN106570654A (en) * | 2016-11-10 | 2017-04-19 | 国网山东省电力公司济南供电公司 | Breakdown rescue real-time situation monitoring and event interactive processing system for power distribution network |
| CN106951564A (en) * | 2017-04-02 | 2017-07-14 | 北京军秀咨询有限公司 | A kind of cloud computing platform analyzed based on data mining and big data and method |
| CN108153815A (en) * | 2017-11-29 | 2018-06-12 | 北京京航计算通讯研究所 | Towards the index classification method of big data |
| CN108459911A (en) * | 2012-06-19 | 2018-08-28 | 微软技术许可有限责任公司 | multi-tenant middleware cloud service technology |
| CN111310042A (en) * | 2020-02-13 | 2020-06-19 | 研祥智能科技股份有限公司 | Collaborative filtering recommendation method and system based on cloud computing |
| CN111431902A (en) * | 2020-03-24 | 2020-07-17 | 深圳市中盛瑞达科技有限公司 | Big data all-in-one machine |
| CN112905323A (en) * | 2021-02-09 | 2021-06-04 | 泰康保险集团股份有限公司 | Data processing method and device, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100083287A1 (en) * | 2008-09-30 | 2010-04-01 | Maximilien E Michael | Declarative Representation of Networked Applications |
| CN101977242A (en) * | 2010-11-16 | 2011-02-16 | 西安电子科技大学 | Layered distributed cloud computing architecture and service delivery method |
-
2011
- 2011-05-16 CN CN 201110125663 patent/CN102169505A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100083287A1 (en) * | 2008-09-30 | 2010-04-01 | Maximilien E Michael | Declarative Representation of Networked Applications |
| CN101977242A (en) * | 2010-11-16 | 2011-02-16 | 西安电子科技大学 | Layered distributed cloud computing architecture and service delivery method |
Non-Patent Citations (1)
| Title |
|---|
| 《广东通信技术》 20101130 林立宇等 基于云计算的电子商务推荐平台的构建分析 第7-10页 1-3 , 第11期 * |
Cited By (32)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103309867A (en) * | 2012-03-09 | 2013-09-18 | 句容智恒安全设备有限公司 | Web data mining system on basis of Hadoop platform |
| CN102663019B (en) * | 2012-03-21 | 2017-06-20 | 北京英孚斯迈特信息技术有限公司 | A kind of instant commending system |
| CN102663019A (en) * | 2012-03-21 | 2012-09-12 | 北京英孚斯迈特信息技术有限公司 | Instant recommendation system |
| CN102724059A (en) * | 2012-03-31 | 2012-10-10 | 常熟市支塘镇新盛技术咨询服务有限公司 | Website operation state monitoring and abnormal detection based on MapReduce |
| CN102724059B (en) * | 2012-03-31 | 2015-03-11 | 常熟市支塘镇新盛技术咨询服务有限公司 | Website operation state monitoring and abnormal detection based on MapReduce |
| CN108459911A (en) * | 2012-06-19 | 2018-08-28 | 微软技术许可有限责任公司 | multi-tenant middleware cloud service technology |
| CN108459911B (en) * | 2012-06-19 | 2022-08-16 | 微软技术许可有限责任公司 | Method and system for multi-tenant middleware cloud service |
| CN103577403A (en) * | 2012-07-19 | 2014-02-12 | 镇江雅迅软件有限责任公司 | Cloud computing technology based recommendation system implementation method |
| CN103812679B (en) * | 2012-11-12 | 2018-01-30 | 深圳中兴网信科技有限公司 | A kind of massive logs statistical analysis system and method |
| CN103812679A (en) * | 2012-11-12 | 2014-05-21 | 深圳中兴网信科技有限公司 | Mass log statistical analysis system and method |
| CN103020282A (en) * | 2012-12-28 | 2013-04-03 | 深圳市彩讯科技有限公司 | General mixed association recommendation development platform and association recommendation method |
| CN103345698A (en) * | 2013-07-09 | 2013-10-09 | 焦点科技股份有限公司 | Personalized recommendation method based on cloud processing mode and applied in e-business environment |
| CN103488775A (en) * | 2013-09-29 | 2014-01-01 | 中国科学院信息工程研究所 | Computing system and computing method for big data processing |
| WO2016008317A1 (en) * | 2014-07-14 | 2016-01-21 | 华为技术有限公司 | Data processing method and central node |
| CN105335135B (en) * | 2014-07-14 | 2019-01-08 | 华为技术有限公司 | Data processing method and central node |
| CN105335135A (en) * | 2014-07-14 | 2016-02-17 | 华为技术有限公司 | Data processing method and center node |
| CN104268709A (en) * | 2014-10-10 | 2015-01-07 | 浪潮集团有限公司 | Method for designing RFID system by distributed LSM tree |
| CN104503967A (en) * | 2014-10-24 | 2015-04-08 | 浪潮电子信息产业股份有限公司 | Hadoop-based network recommendation method |
| CN104834557A (en) * | 2015-05-18 | 2015-08-12 | 成都博元科技有限公司 | Data analysis method based on Hadoop |
| CN106294439A (en) * | 2015-05-27 | 2017-01-04 | 北京广通神州网络技术有限公司 | A kind of data recommendation system and data recommendation method thereof |
| CN105321124A (en) * | 2015-11-23 | 2016-02-10 | 南京信息工程大学 | Hadoop-based electric power cloud platform design scheme |
| CN105677846B (en) * | 2016-01-06 | 2019-12-31 | 中国传媒大学 | Recommendation system and construction method thereof |
| CN105677846A (en) * | 2016-01-06 | 2016-06-15 | 中国传媒大学 | Recommendation system and construction method thereof |
| CN106503140A (en) * | 2016-10-20 | 2017-03-15 | 安徽大学 | One kind is based on Hadoop cloud platform web resource personalized recommendation system and method |
| CN106570654A (en) * | 2016-11-10 | 2017-04-19 | 国网山东省电力公司济南供电公司 | Breakdown rescue real-time situation monitoring and event interactive processing system for power distribution network |
| CN106339829A (en) * | 2016-11-10 | 2017-01-18 | 国网山东省电力公司济南供电公司 | Big data, Cloud, IoT and mobile internet technologies based active maintenance panorama monitoring system of power distribution network |
| CN106951564A (en) * | 2017-04-02 | 2017-07-14 | 北京军秀咨询有限公司 | A kind of cloud computing platform analyzed based on data mining and big data and method |
| CN108153815A (en) * | 2017-11-29 | 2018-06-12 | 北京京航计算通讯研究所 | Towards the index classification method of big data |
| CN111310042A (en) * | 2020-02-13 | 2020-06-19 | 研祥智能科技股份有限公司 | Collaborative filtering recommendation method and system based on cloud computing |
| CN111431902A (en) * | 2020-03-24 | 2020-07-17 | 深圳市中盛瑞达科技有限公司 | Big data all-in-one machine |
| CN112905323A (en) * | 2021-02-09 | 2021-06-04 | 泰康保险集团股份有限公司 | Data processing method and device, electronic equipment and storage medium |
| CN112905323B (en) * | 2021-02-09 | 2023-10-27 | 泰康保险集团股份有限公司 | Data processing method, device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102169505A (en) | Recommendation system building method based on cloud computing | |
| He et al. | A state-of-the-art survey of cloud manufacturing | |
| Fernández et al. | Big Data with Cloud Computing: an insight on the computing environment, MapReduce, and programming frameworks | |
| Hwang | Cloud computing for machine learning and cognitive applications | |
| Zeng et al. | An integrated task computation and data management scheduling strategy for workflow applications in cloud environments | |
| Yuan et al. | A highly practical approach toward achieving minimum data sets storage cost in the cloud | |
| CN104081353A (en) | Dynamic load balancing in a scalable environment | |
| CN105809356A (en) | Information system resource management method based on application integrated cloud platform | |
| CN102521014A (en) | Deploying method and deploying device for virtual machine | |
| CN101986661A (en) | Improved MapReduce data processing method under virtual machine cluster | |
| CN103310460A (en) | Image characteristic extraction method and system | |
| Huang | The Workflow Task Scheduling Algorithm Based on the GA Model in the Cloud Computing Environment. | |
| Tao et al. | Job scheduling optimization for multi-user MapReduce clusters | |
| Qureshi et al. | A comparative analysis of resource allocation schemes for real-time services in high-performance computing systems | |
| CN106056322A (en) | Smart grid scheduling system based on cloud computing | |
| CN104050193A (en) | Message generating method and data processing system for realizing method | |
| CN111144830A (en) | Enterprise-level computing resource management method, system and computer equipment | |
| He et al. | Haas: Cloud-based real-time data analytics with heterogeneity-aware scheduling | |
| CN107528871A (en) | Data analysis in storage system | |
| Kim et al. | Optimized data processing analysis using big data cloud platform | |
| CN107274135A (en) | The raw tobacco material overall planning method and system shared based on cooperative information | |
| Gong | Workflow scheduling based on mobile cloud computing machine learning | |
| Talei et al. | An end to end real time architecture for analyzing and clustering time series data: case of an energy management system | |
| Murugesan et al. | Multi-source task scheduling in grid computing environment using linear programming | |
| Khan | Hadoop performance modeling and job optimization for big data analytics |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20110831 |