CN106127546A - A kind of Method of Commodity Recommendation based on the big data in intelligence community - Google Patents

Abstract

The invention discloses a kind of Method of Commodity Recommendation based on the big data in intelligence community, relate to internet information processing technology field, comprise the steps: first, carry out data prediction；Secondly, sample set structure is carried out；Again, Feature Engineering process is carried out；Then, Model Fusion is carried out；Finally, personalized commercial information is recommended for user；Wherein, sample set is configured to: extracts the one-level interbehavior sample in nearest n days and second order behavior sample, extracts the one-level behavior sample in nearest n+m days, and form sampling sample；The feature of sample set data comprises foundation characteristic, cross feature.The present invention has the beneficial effect that and reduces negative sample at the content of sample set, improves commodity purchasing predictablity rate, provides the user more accurately, more believable commercial product recommending service.Meanwhile, use cross feature for describing the data characteristics of sample set, be used for predicting the user behavior of specific meanings.

Description

A kind of Method of Commodity Recommendation based on the big data in intelligence community

Technical field

The present invention relates to internet information processing technology field, particularly relate to a kind of commodity based on the big data in intelligence community Recommendation method.

Background technology

Along with developing rapidly of computer technology and ecommerce, commodity number and kind quickly increase, and user the most more comes More hanker after shopping online.And in urbanization process, development smart city has been increasingly becoming the main trend of urban construction, During this, as the important component part of smart city, intelligence community have also been obtained to be greatly developed.Therefore, exist along with user Shopping ratio in the Internet increases year by year, how in intelligence community rapid and convenient, accurately carry out commercial product recommending for user and become For insider and brainstrust focus of attention.

In the face of a feast for the eyes commodity, user requires a great deal of time and just can find the commodity self wanting to buy.For Solving this problem, personalized recommendation system arises at the historic moment.Under real business scenario, generally require all of commodity Subset build Personalization recommendation model.During completing this part task, not only need to utilize user this business Behavioral data in product subset, also needs to utilize more rich user behavior data etc. simultaneously.Accordingly, it would be desirable to effectively build individual Property commending system excavate from these mass users purchase datas, analyze, integrate out useful data, provide the user completely The decision support of property and information service.The most common proposed algorithm mainly has based on commending contents, pushes away based on collaborative filtering Recommend and knowledge based is recommended.Directly perceived based on its recommendation results of commending contents, easily resolve, it is not necessary to domain knowledge, but right The most disposable in complex properties, and need enough data configuration graders；Based on collaborative filtering recommending, it has recommendation individual character Change, automaticity height, it is possible to process the advantages such as complicated destructuring object, but not can solve new user and ask Topic, new projects' problem, openness problem, and system extensibility problem；Knowledge based is recommended to can be considered one to a certain extent Planting inference technology, it is not built upon user needs and recommendation on the basis of preference.Owing to various recommendation methods have pluses and minuses, Therefore, in reality, combined recommendation is often used.I.e. by avoiding to a certain extent or make up and each recommend skill after combination The weakness of art.

Based under this background, the present invention proposes a kind of Method of Commodity Recommendation based on the big data in intelligence community, available The big data of intelligence community, in conjunction with community users behavior, carry out service recommendation to user the Internet.By data prediction, right Abnormal data and hash are purged.For uneven situation, use classical over-sampling algorithm or by double for positive sample, or Use lack sampling, reduce part negative sample.Onestep extraction of going forward side by side goes out foundation characteristic and cross feature, uses XGBoost multi-model Merge and obtain consequently recommended result.The present invention has higher Stability and veracity, uses Biodiversity Characteristics to merge skill simultaneously Art, can improve accuracy rate and the recall rate of recommendation further, can be that net purchase user provides more accurately, more believable commercial product recommending takes Business.

Summary of the invention

Because the drawbacks described above of prior art, the technical problem to be solved is to provide a kind of based on wisdom society The Method of Commodity Recommendation of the big data in district, provides the user more accurately, more believable commercial product recommending service.On the one hand, to the most equal Level interbehavior, sets the time range of different sample collections, is built into sample set, reduces negative sample containing at sample set Amount.On the other hand, the present invention also uses cross feature to be used for describing the data characteristics of sample set, for predicting the use of specific meanings Family behavior, buys a large amount of different article under similar commodity, dopes commodity that user wants most to buy, pre-such as one user of prediction Measure the classification that user wants most to buy.

For achieving the above object, the invention provides a kind of Method of Commodity Recommendation based on the big data in intelligence community, it is special Levy and be, comprise the steps:

S1, data prediction: remove abnormal data and hash, form initial sample；

S2, sample set build: be used for extracting initial sample, build sample set；

S3, Feature Engineering: for extracting the feature of sample set data；

S4, Model Fusion: use the feature construction model extracted, and carry out Model Fusion；

S5, result are recommended: recommend personalized commercial information for user；

Wherein, in described step S2, described sample set is configured to: extract one-level interbehavior sample in nearest n days and Second order behavior sample, extracts the one-level behavior sample in nearest n+m days, and forms sampling sample；Wherein, n >=0, m >=0；Described In step S3, the feature of described sample set data comprises foundation characteristic, cross feature.

In this technical scheme, data prediction has the beneficial effects that: effectively reduce sample size；In sample set builds, One-level interbehavior sample, the extraction time scope difference of two grades of interbehavior samples, have the beneficial effects that: reduce negative further Sample size；In Feature Engineering, comprise foundation characteristic, cross feature, have the beneficial effects that: the intersection providing specific meanings is special Levy, for predicting the user behavior of specific meanings, buy a large amount of different article, the predictions under similar commodity such as one user of prediction Go out user to want most the commodity bought, dope the classification that user wants most to buy.

Furthermore, one-level interbehavior sample includes: the collection behavior sample of commodity, add shopping cart behavior sample, Purchasing behavior sample；Described two grades of interbehavior samples include: the navigation patterns sample of commodity；

The technical program has the beneficial effect that and commodity carry out collection behavior, adds the use of shopping cart behavior, purchasing behavior Family, has bigger probability to buy this commodity, and sets it to one-level interbehavior sample；To the purchase probability of goods browse relatively Little, it is set as two grades of interbehavior samples.

Furthermore, foundation characteristic includes user characteristics, product features, category feature；Described cross feature includes using Family product features, class of subscriber feature, merchandise classification feature and the cross feature of three；

Furthermore, in described step S3, also include conversion and the smoothing processing of described sample data；Described change Change and comprise the steps: with smoothing processing

S31 data convert: according to formula X=ln (1+x), convert characteristic；Wherein, x is data before conversion, Described X is data after conversion；

S32 smoothing processing: data after conversion are smoothed；

The technical program uses and characteristic carries out smoothing techniques, thus weakens the impact of abnormal data；Smoothing Process and not only include, the smoothing processing after the sample data disturbed in step s3 is deleted, also include in step sl The smoothing processing of the abnormal data deleted, the smooth place of large-scale joint data vacation such as the most deleted double 11, double 12 Reason.

Furthermore, Model Fusion is to use the XGBoost model feature to extracting to carry out importance ranking, goes forward side by side Row packet, then carry out XGBoost model and averagely merge.

The technical program is beneficial in that, it is to avoid over-fitting, it is ensured that the stability of model.

Furthermore, in described step S5, including:

S51, accuracy rate Precision of checking proposed algorithm and recall rate Recall；

The threshold value that S52, adjustment predict the outcome, makes F1 value maximum, and obtains recommendation results；

S53, according to described recommendation results, recommend personalized commercial information for user；

Wherein, PrecesionSet is the purchase data acquisition system of algorithm predicts, and ReferenceSet is that true answer is bought Data acquisition system；

Described

Described

Described

The technical program has the beneficial effect that the effectiveness of checking proposed algorithm, adjusts the threshold value predicted the outcome and takes recommendation As a result, accuracy rate and recall rate is made to reach balance.

The invention has the beneficial effects as follows: a kind of Method of Commodity Recommendation based on the big data in intelligence community is provided, carries for user For more accurately, more believable commercial product recommending service.On the one hand, to different brackets interbehavior, different sample collections is set Time range, is built into sample set, reduces the negative sample content at sample set.On the other hand, the present invention also uses cross feature For describing the data characteristics of sample set, for predicting the user behavior of specific meanings, such as prediction, one user buys similar business A large amount of different article under product, dope user and want most the commodity bought, dope the classification that user wants most to buy.

Accompanying drawing explanation

Proposed algorithm flow chart is moved in the intelligence community that Fig. 1 provides for the embodiment of the present invention one；

The samples selection figure that Fig. 2 provides for the embodiment of the present invention one；

The purchase conversion ratio figure that Fig. 3 provides for the embodiment of the present invention one；

Fig. 4 is investigation day negative sample scattergram for the first n days interactive objects that the embodiment of the present invention one provides；

The foundation characteristic figure that Fig. 5 provides for the embodiment of the present invention one；

The cross feature figure that Fig. 6 provides for the embodiment of the present invention one；

The feature selection approach comparison diagram that Fig. 7 provides for the embodiment of the present invention one；

Fig. 8 for the embodiment of the present invention one provide based on XGBoost proposed algorithm flow chart.

Detailed description of the invention

The invention will be further described with embodiment below in conjunction with the accompanying drawings:

As it is shown in figure 1, a kind of based on the big data in intelligence community the Method of Commodity Recommendations that Fig. 1 provides for the embodiment of the present invention Flow chart, specifically include:

S1, data prediction: remove abnormal data and hash, form initial sample；

S2, sample set build: be used for extracting initial sample, build sample set；

S3, Feature Engineering: for extracting the feature of sample set data；

S4, Model Fusion: use the feature construction model extracted, and carry out Model Fusion；

S5, result are recommended: recommend personalized commercial information for user；

In the present embodiment, the main electricity using intelligence community is purchased the big data of thing and discusses.In the present embodiment, just Beginning data contain the sampling a certain amount of user out behavioral data within one month, table 1 gives user Behavioral data definition and explanation.In table 2, commodity subset comprises the commodity sign for buying prediction and commodity classification mark Know.Wherein, user behavior is divided into 4 kinds: " 1 " browses；" 2 " collect；" 3 " add shopping cart；" 4 " are bought.

Table 1 user behavioral data on commodity complete or collected works

Table 2 is for buying the commodity subset of prediction

In the present embodiment, need to use primary data to set up recommended models, export user at following one day to business Predicting the outcome of product subset purchasing behavior.I.e. provide a certain amount of user behavioral data within 30 days, it was predicted that user is the 31st It to purchase data, it was predicted which user can buy which kind of commodity in commodity subset.

Each step is described in detail below.

S1, data prediction: remove abnormal data and hash, form initial sample；

In the present embodiment, owing to data exist substantial amounts of abnormal data and hash, and the existence of these data can be produced Raw Expired Drugs, it is therefore desirable to this part sample is rejected.Based on this, the present embodiment has mainly deleted a large amount of operation but Large-scale joint data vacation such as all data of the user never bought and seldom buy, especially deletion double 11, double 12.For number According to uneven situation, use classical over-sampling algorithm or by double for positive sample, or use lack sampling, reduce the negative sample of part This.In the present embodiment, mainly have employed positive sample double, negative sample stochastical sampling, and keep the ratio of 1:10；Finally, formed Initial sample.

S2, sample set build: be used for extracting initial sample, build sample set；

In this embodiment, sample set construction method is: extract the one-level interbehavior sample in nearest n days and second order behavior Sample, extracts the one-level behavior sample in nearest n+m days, and forms sampling sample；Wherein, n >=0, m >=0；One-level interbehavior Sample includes: the collection behavior sample of commodity, add shopping cart behavior sample, purchasing behavior sample；Two grades of interbehavior sample packages Include: the navigation patterns sample of commodity；

The mutual sample of one-level that specifically, in the present embodiment, sampling sample includes investigating in 2-7 days a few days ago (collection, Add shopping cart, purchase) and all mutual sample (browse, collect, add shopping cart, purchase) that occurs the previous day build sample Collection.It is illustrated in figure 2 the present embodiment samples selection figure.

If Fig. 3 is that the embodiment of the present invention buys conversion ratio figure.If Fig. 4 is that first n days interactive objects divide for investigating day negative sample Cloth.From Fig. 3, data understand, and the conversion ratio of buying browsing, collect, add shopping cart is similar distribution, is investigating 7-10 a few days ago It sample is bought conversion ratio and is tended to 0.But, to observe according to data in Fig. 4 and draw, the sample browsed increases on foot, pushes away the most forward one The negative sample that it is not bought increases by 200,000,000.

If all users combining all commodity as sample set, do not hand over negative sample amount huge, although recall rate is high, but Accurate rate is low, predicts purchased sample just as looking for a needle in a haystack in great amount of samples；If only considering to investigate the mutual sample of day, Then can miss the sample just bought after interbehavior occurs many days, recall rate is low.So, the present embodiment is from conversion ratio, sample Amount, model performance efficiency consider, non-in 2-7 days before extracting browse owning of occurring (collect, add shopping cart, purchase) and the previous day Mutual sample builds sample set.

S3, Feature Engineering: for extracting the feature of sample set data；

In the present embodiment, sample set data are carried out feature extraction.The feature of sample set data comprises foundation characteristic, friendship Fork feature.It is respectively embodiment of the present invention foundation characteristic and cross feature figure with reference to such as Fig. 5 and Fig. 6.Foundation characteristic includes user Feature, product features, and category feature.Cross feature includes that user's product features, class of subscriber feature, merchandise classification are special Levy, and three's cross feature.Referring to table 3 in the embodiment of the present invention, table 3 show cross feature implication table.

3 two statistical natures of table

Characteristic statistics window, in order to fully construct multifarious foundation characteristic and cross feature, is divided by the present embodiment It is 1/2/3/4/5/6/7/10/15/20/30, is i.e. respectively directed to user which sky in month and carries out characteristic statistics.And to spy Levy and be smoothed, thus construct Biodiversity Characteristics.

Conversion and the smoothing processing of described sample data is also included, including walking as follows in step S3 of the present embodiment Rapid:

S31 data convert: according to formula X=ln (1+x), convert characteristic；Wherein, x is data before conversion, Described X is data after conversion；

S32 smoothing processing: data after conversion are smoothed.

In the present embodiment, feature smoothing processing mainly uses ln (1+x) function, ratio division is converted into subtraction, goes forward side by side Row data smoothing standardization, thus weaken the impact of abnormal data.

In order to eliminate or weaken the impact of interference, improve the smoothness of curve, characteristic must be smoothed.Number It is to eliminate the interference component in data according to the rule of smoothing processing, keeps the variation characteristic of original curve again.Often Smoothing processing method have: averaging method, splines method and five-spot triple smoothing etc..Averaging method is relatively simple, filter Ripple effect is the most poor；Splines method utilizes spline interpolation to approach the method for sampled point to realize smothing filtering, and algorithm is various, effect Fruit is preferably；Five-spot triple smoothing utilizes polynomial least square approximation that sampled point realizes smothing filtering, and algorithm is simple, Effect is preferable.Preferably, use five-spot triple smoothing that characteristic is smoothed in the present embodiment.

It is noted that in this embodiment, smoothing techniques not only includes, to the sample data disturbed in step s3 Smoothing processing after deletion, also includes the smoothing processing to the abnormal data deleted, the most deletes The smoothing processing of large-scale joint data vacation such as double 11, double 12 removed.

S4, Model Fusion, use the feature construction model extracted, and carry out Model Fusion.

In the present embodiment, sample set is randomly divided into training sample set test sample collection.In the present embodiment, model The input of blending algorithm is: training sample set TrainSet, test sample collection TestSet, complete or collected works feature space F, total characteristic number L, feature divides space number k.Model Fusion algorithm is output as: probability P i that each sample is purchased.Specifically, model melts Conjunction comprises the steps of:

Step S41: use XGBoost training pattern, and export each feature space score value score_j, and be expressed as score_j=score (xgb_train (TrainSet))；Wherein j=0,1 ... l-1；

Step S42: feature score sequence delivery are obtained the dividing subset space F of feature_i, and it is expressed as F_i= Rank (score) %k；Wherein i=0,1 ... k-1；

Step S43: each feature space is trained prediction, it is thus achieved that training pattern Model_i, and is expressed as Model_i=Train (Fi)；Wherein i=0,1 ... k-1

Step S44: k training pattern is carried out Model Fusion, and is expressed asMeanwhile, adjust The whole threshold value that predicts the outcome filtersOutput.

As shown in Figure 7 and Figure 8, Fig. 7 is feature selection approach comparison diagram in the embodiment of the present invention, and Fig. 8 is that the present invention implements Based on XGBoost proposed algorithm flow chart in example.Specifically, the present embodiment be based on primitive character generate foundation characteristic and Cross feature, thus constructed high-dimensional feature collection.On the one hand high dimensional feature may result in dimension disaster, the most then It is easily caused over-fitting, it is therefore desirable to do dimension-reduction treatment.

The most common dimension reduction method has t-SNE, PCA.Wherein t-SNE computation complexity is high, and PCA then needs hypotheses Data are Gauss distribution.In addition to using dimension-reduction algorithm, it would however also be possible to employ feature selection reduces characteristic dimension.Common spy The method levying selection mainly has maximum information coefficient, Pearson correlation coefficients, regularization method, feature ordering side based on model Method.Wherein feature ordering method based on learning model is the more efficient algorithm of one, it is advantageous that the mistake of model learning In journey, the process with feature selection is carried out simultaneously, therefore mainly have employed this kind of method in the present embodiment.

Algorithm based on decision tree, such as random forest, boosted tree, can be defeated after model training completes Go out the importance of feature, therefore carry out feature selection for the data acquisition XGBoost in the present embodiment.XGBoost is A kind of realization of boosted tree, efficiency and precision are the highest.And XGBoost is that a kind of parallel GBDT realizes, different In traditional GBDT mode, only make use of the derivative information of single order, and loss function has been done the Thailand of second order by XGBoost Strangle and launch, and outside object function, add regular terms entirety seek optimal solution, in order to weigh decline and the model of object function Complexity, it is to avoid over-fitting.

In the present embodiment, carry out feature importance ranking based on XGBoost, choose k feature of top, and feature is discrete Change to 10 intervals, thus obtain 10 stack features subset space f_i, wherein i ∈ [0,9].XGBoost multi-model is finally used to put down All merge, thus avoid over-fitting, it is ensured that the stability of model.

S5, result are recommended: recommend personalized commercial for user.

Specifically, in step S5 of the present embodiment, including:

S51, accuracy rate Precision of checking proposed algorithm and recall rate Recall；

The threshold value that S52, adjustment predict the outcome, makes F1 value maximum, and obtains recommendation results；

S53, according to described recommendation results, recommend personalized commercial information for user；

In the present embodiment step S5, by initial data or test data, through data prediction and feature extraction, and generation Enter model training.Adjusting the threshold value that predicts the outcome and take recommendation results, make accuracy rate and recall rate reach balance, namely F1 value is Greatly.

In order to verify the effectiveness of proposed algorithm in the present embodiment, use classical accuracy rate (precision), recall rate (recall), and F1 value is as evaluation index, and its computing formula is as follows:

$\mathrm{Pr}ecision=\frac{|\∩(\mathrm{Pr}ecisionSet,\mathrm{Re}ferenceSet)|}{\left|\mathrm{Pr}ecisionSet\right|}$

$\mathrm{Re}call=\frac{|\∩(\mathrm{Pr}ecisionSet,\mathrm{Re}ferenceSet)|}{\left|\mathrm{Re}ferenceSet\right|}$

$F1=\frac{2\×\mathrm{Pr}ecision\×\mathrm{Re}call}{\mathrm{Pr}ecision+\mathrm{Re}call}$

Wherein PrecesionSet is the purchase data acquisition system of algorithm predicts, and ReferenceSet is that number is bought in true answer According to set, using F1 value as final unique evaluating standard.

The preferred embodiment of the present invention described in detail above.Should be appreciated that those of ordinary skill in the art without Need creative work just can make many modifications and variations according to the design of the present invention.Therefore, all technology in the art Personnel are available by logical analysis, reasoning, or a limited experiment the most on the basis of existing technology Technical scheme, all should be in the protection domain being defined in the patent claims.

Claims (6)

1. a Method of Commodity Recommendation based on the big data in intelligence community, it is characterised in that comprise the steps:

S1, data prediction: remove abnormal data and hash, form initial sample；

S2, sample set build: be used for extracting initial sample, build sample set；

S3, Feature Engineering: for extracting the feature of sample set data；

S4, Model Fusion: use the feature construction model extracted, and carry out Model Fusion；

S5, result are recommended: recommend personalized commercial information for user；

Wherein, in described step S2, described sample set is configured to: extract the one-level interbehavior sample in nearest n days and two grades Behavior sample, extracts the one-level behavior sample in nearest n+m days, and forms sampling sample；Wherein, n >=0, m >=0；Described step In S3, the feature of described sample set data comprises foundation characteristic, cross feature.

A kind of Method of Commodity Recommendation based on the big data in intelligence community, it is characterised in that described one Level interbehavior sample includes: the collection behavior sample of commodity, add shopping cart behavior sample, purchasing behavior sample；Described two grades Interbehavior sample includes: the navigation patterns sample of commodity.

A kind of Method of Commodity Recommendation based on the big data in intelligence community, it is characterised in that described base Plinth feature includes user characteristics, product features, category feature；Described cross feature includes that user's product features, class of subscriber are special Levy, merchandise classification feature and the cross feature of three.

A kind of Method of Commodity Recommendation based on the big data in intelligence community, it is characterised in that described Step S3 also includes conversion and the smoothing processing of described sample data；Described conversion comprises the steps: with smoothing processing

S31 data convert: according to formula X=ln (1+x), convert characteristic；Wherein, x is data before conversion, described X is data after conversion；

S32 smoothing processing: data after conversion are smoothed.

A kind of Method of Commodity Recommendations based on the big data in intelligence community the most according to claim 1, it is characterised in that described Model Fusion is to use the XGBoost model feature to extracting to carry out importance ranking, and is grouped, then carries out XGBoost model averagely merges.

A kind of Method of Commodity Recommendations based on the big data in intelligence community the most according to claim 1, it is characterised in that in institute State in step S5, including:

S51, accuracy rate Precision of checking proposed algorithm and recall rate Recall；

The threshold value that S52, adjustment predict the outcome, makes F1 value maximum, and obtains recommendation results；

S53, according to described recommendation results, recommend personalized commercial information for user；

Wherein, PrecesionSet is the purchase data acquisition system of algorithm predicts, and ReferenceSet is that data are bought in true answer Set；

Described

Described

Described