CN103544216A - Information recommendation method and system combining image content and keywords - Google Patents

Abstract

The invention discloses an information recommendation method and system combining image content and keywords. The information recommendation method combining image content and keywords comprises the steps that keyword information of images in an image library and image content information containing color features and textural features are extracted, the keyword information and the image content information are expressed as a vector space model, and a corresponding keyword information matrix and an image content information matrix are obtained; the keyword information matrix and the image content information matrix are processed by utilizing a linear sparse model, and a similarity chart is obtained by calculating the similarity among the images; an image similar to a target image is inquired from the similarity chart according to the target image searched by a user, and an original recommendation list is formed; the original recommendation list is arranged to obtain a final recommendation list, and the final recommendation list is displayed.

Description

Information recommendation method and the system of a kind of combining image content and key word

Technical field

The present invention relates to image retrieval and information recommendation field, relate in particular to information recommendation method and the system of a kind of combining image content and key word.

Background technology

High speed development along with internet and ecommerce, online multimedia messages also explosion type increase, so people are also more and more to the Search Requirement of multimedia messages, for example shopping online, social networks, video sharing platform etc. all needs multimedia messages to retrieve.Multimedia messages on internet generally includes image, Voice & Video.Wherein the retrieval of image is particularly focus and the trend of ecommerce of internet, applications.For example, thereby only retrieve relevant commodity by picture self-information, facilitate user to inquire about on the one hand, help system is made recommendation to user effectively on the other hand.

The retrieval of internet epigraph has the direction of two main flows: first is to take key word as basic retrieval (Keyword-Based Retrieval); Second is that to take the content of image self be basic retrieval (Content-Based Retrieval).

, Google image search engine etc.There is the problem of two aspects in this method: the one, and this method needs more artificial participation, and along with the increase of picture number, this method is difficult to realize; Second Problem is that the quantity of information that image comprises is huge, different people is not identical for the understanding of same image yet, this just causes the unified standard of the mark neither one of image, and light is to be difficult to accomplish describe exactly and retrieving multimedia information with key word.

Information retrieval based on picture material is that the content of multimedia object and context semantic environment are retrieved, as the color in image, texture, or the scene in video, segment is analyzed and feature extraction, and carry out similarity matching based on these features, the system of information retrieval based on contents has the QBIC of IBM exploitation, VisualSEEK image query systems of the Photobook of MIT multi-media Laboratory, Columbia Univ USA's exploitation etc.Content-based information retrieval has stronger objectivity, and still, due to these features real semantic information of representative image not, content-based retrieval result is often unsatisfactory, so most system or the image retrieval based on key word.

No matter be all Shortcomings of image indexing system based on key word or the image indexing system based on picture material.Therefore, prior art has yet to be improved and developed.

Summary of the invention

In view of above-mentioned the deficiencies in the prior art, the object of the present invention is to provide information recommendation method and the system of a kind of combining image content and key word, be intended to solve that conventional images searching system or workload are large, the inaccurate problem of result for retrieval.

Technical scheme of the present invention is as follows:

An information recommendation method for combining image content and key word, wherein, comprises step:

A, extract picture library picture keyword message and comprise color characteristic and the image content information of textural characteristics, keyword message and image content information are expressed as to vector space model, obtain corresponding keyword message matrix and image content information matrix;

B, utilize Sparse model to process keyword message matrix and image content information matrix, the similarity in calculating chart valut between picture obtains similar kilsyth basalt;

C, according to the Target Photo of user search, from similar kilsyth basalt, inquire about the similar pictures of described Target Photo and form original recommendation list;

D, arrange original recommendation list and obtain final recommendation list and show.

Described combining image content and the information recommendation method of key word, wherein, in described steps A, the process of extracting the image content information that comprises color characteristic and textural characteristics specifically comprises:

By picture according to preassigned quantity piecemeal;

Each piecemeal is processed and obtained the histogram that comprises several minizones by color cell, and each minizone represents a kind of color;

The textural characteristics at abstract image edge;

To the color characteristic extracting with textural characteristics quantizes and Regularization, obtain the CEDD histogram of every pictures.

Described combining image content and the information recommendation method of key word, wherein, the algorithmic formula of the sparse model of described step B neutral line is as follows:

$\begin{array}{cc}\underset{\mathrm{<figure-callout\; id="1"\; label="S"\; filenames="HDA00003858225300041.png"\; state="\{\{state\}\}">S</figure-callout>}}{\mathrm{Minimize}}& \frac{1}{2}{\left|\right|M-\mathrm{MS}\left|\right|}_F^2+\frac{\mathrm{\&alpha;}}{2}\left|\right|{F-\mathrm{FS}\left|\right|}_F^2+\frac{\mathrm{\&beta;}}{2}{\left|\right|S\left|\right|}_F^2+\mathrm{\&lambda;}{\left|\right|S\left|\right|}_1,\end{array}$ $s.t\begin{array}{c}S\&GreaterEqual;0,\\ \mathrm{diag}\left(S\right)=0.\end{array};$

Wherein, M, F represent respectively keyword message matrix (m _ij) _{m * n}with image content information matrix (f _ij) _{p * n}, S represents similarity matrix between calculative picture, || || _f, || || ₁represent respectively Frobenius norm and 1-norm, m, p represent respectively key characteristics quantity and the image content features quantity of all pictures, and n represents picture number, the factor of influence of α representative image content information matrix, and β, λ are regularization parameter.

Described combining image content and the information recommendation method of key word, wherein, be normalized F:

$F(i,j)=\frac{F(i,j)-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}}{\mathrm{Max}\left\{F(i,\&CenterDot;)\right\}-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}};$

Wherein, capable, the j column element of F (i, j) representing matrix F i; Min{F (i) } minimum value of the capable all elements of representing matrix F i; Max{F (i) } maximal value of the capable all elements of representing matrix F i, 0≤i<p, 0≤j<n.

Described combining image content and the information recommendation method of key word, wherein, described step B also comprises:

When having new picture to be added into picture library, adopt the computing formula of sparse linear model incremental update to calculate the similarity between other pictures in new picture and picture library, upgrade sparse linear model.

Described combining image content and the information recommendation method of key word, wherein, the step that the original recommendation list of described arrangement obtains final recommendation list displaying further comprises:

When obtaining a plurality of original recommendation list, the picture in all original recommendation list is filtered, and the picture after filtering is sorted and obtains final recommendation list.

Described combining image content and the information recommendation method of key word, wherein, the step that the original recommendation list of described arrangement obtains final recommendation list displaying specifically comprises:

Picture in original recommendation list is filtered and sorted obtain final recommendation list, show described final recommendation list.

Described combining image content and the information recommendation method of key word, wherein, the described step of inquiring about the similar pictures of described Target Photo according to the Target Photo of user search and forming original recommendation list from similar kilsyth basalt specifically comprises:

When user search picture, obtain the ID of the Target Photo of user search, according to the ID of Target Photo, from similar kilsyth basalt, inquire the similar pictures relevant to Target Photo and form original recommendation list.

An information recommendation system for combining image content and key word, wherein, comprising:

Matrix conversion module, for extract picture library picture keyword message and comprise color characteristic and the image content information of textural characteristics, keyword message and image content information are expressed as to vector space model, obtain corresponding keyword message matrix and image content information matrix;

Similarity calculation module, for utilizing Sparse model to process keyword message matrix and image content information matrix, the similarity in calculating chart valut between picture obtains similar kilsyth basalt;

Original recommendation list acquisition module, for inquiring about the similar pictures of described Target Photo and form original recommendation list from similar kilsyth basalt according to the Target Photo of user search;

Final recommendation list acquisition module, obtains final recommendation list and shows for arranging original recommendation list.

Described combining image content and the information recommendation system of key word, wherein, described matrix conversion module comprises:

Minute module unit, for by picture according to preassigned quantity piecemeal;

Color processing unit, for each piecemeal is processed and obtained the histogram that comprises several minizones by color cell, each minizone represents a kind of color;

Texture processing unit, for the textural characteristics at abstract image edge;

CEDD histogram acquiring unit, for to the color characteristic extracting with textural characteristics quantizes and Regularization, obtains the CEDD histogram of every pictures.

Described combining image content and the information recommendation system of key word, wherein, described similarity calculation module comprises:

Algorithm unit, for according to the similarity between keyword message matrix and image content information matrix computations picture, wherein algorithmic formula is as follows:

$\begin{array}{cc}\underset{\mathrm{<figure-callout\; id="1"\; label="S"\; filenames="HDA00003858225300041.png"\; state="\{\{state\}\}">S</figure-callout>}}{\mathrm{Minimize}}& \frac{1}{2}{\left|\right|M-\mathrm{MS}\left|\right|}_F^2+\frac{\mathrm{\&alpha;}}{2}\left|\right|{F-\mathrm{FS}\left|\right|}_F^2+\frac{\mathrm{\&beta;}}{2}{\left|\right|S\left|\right|}_F^2+\mathrm{\&lambda;}{\left|\right|S\left|\right|}_1,\end{array}$ $s.t\begin{array}{c}S\&GreaterEqual;0,\\ \mathrm{diag}\left(S\right)=0.\end{array};$

Wherein, M, F represent respectively keyword message matrix (m _ij) _{m * n}with image content information matrix (f _ij) _{p * n}, S represents similarity matrix between calculative picture, || || _f, || || ₁represent respectively Frobenius norm and 1-norm, m, p represent respectively key characteristics quantity and the image content features quantity of all pictures, and n represents picture number, the factor of influence of α representative image content information matrix, and β, λ are regularization parameter.

Described combining image content and the information recommendation system of key word, wherein, described similarity calculation module comprises:

Normalized unit, for matrix F is normalized:

$F(i,j)=\frac{F(i,j)-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}}{\mathrm{Max}\left\{F(i,\&CenterDot;)\right\}-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}};$

Wherein, capable, the j column element of F (i, j) representing matrix F i; Min{F (i) } minimum value of the capable all elements of representing matrix F i; Max{F (i) } maximal value of the capable all elements of representing matrix F i, 0≤i<p, 0≤j<n.

Described combining image content and the information recommendation system of key word, wherein, similarity calculation module also comprises:

Incremental computations unit, for when having new picture to be added into picture library, adopts the computing formula of sparse linear model incremental update to calculate the similarity between other pictures in new picture and picture library, upgrades sparse linear model.

Beneficial effect: the present invention calculates the similarity between picture with Sparse models coupling keyword message and image content information, comprising with CEDD model, picture material being carried out to feature extraction.Strong adaptability of the present invention, the recommendation request that meets user that can be real-time, simultaneously will be generally higher than being used single model in the effect of recommending, and wherein the amplification in this index of MRR can approach 100%.

Accompanying drawing explanation

Fig. 1 is the process flow diagram of the information recommendation method preferred embodiment of combining image content of the present invention and key word.

Fig. 2 is the system chart of the online part of the information recommendation system of combining image content of the present invention and key word and off-line part.

Fig. 3 is the particular flow sheet of step S101 in method shown in Fig. 1.

Fig. 4 is the system chart of the information recommendation system preferred embodiment of combining image content of the present invention and key word.

Fig. 5 is the concrete structure block diagram of matrix conversion module in system shown in Figure 4.

Fig. 6 is prec@n and recall@n index design sketch in the recommendation effect of employing information recommendation system of the present invention.

Fig. 7 is MAP, NDCG, MRR and AUC index design sketch in the recommendation effect of employing information recommendation system of the present invention.

Embodiment

The invention provides information recommendation method and the system of a kind of combining image content and key word, for making object of the present invention, technical scheme and effect clearer, clear and definite, below the present invention is described in more detail.Should be appreciated that specific embodiment described herein, only in order to explain the present invention, is not intended to limit the present invention.

Refer to Fig. 1, Fig. 1 is the process flow diagram of the information recommendation method preferred embodiment of combining image content of the present invention and key word, and as shown in the figure, it comprises:

S101, extract picture library picture keyword message and comprise color characteristic and the image content information of textural characteristics, keyword message and image content information are expressed as to vector space model, obtain corresponding keyword message matrix and image content information matrix;

S102, utilize Sparse model to process keyword message matrix and image content information matrix, the similarity in calculating chart valut between picture obtains similar kilsyth basalt;

S103, according to the Target Photo of user search, from similar kilsyth basalt, inquire about the similar pictures of described Target Photo and form original recommendation list;

S104, arrange original recommendation list and obtain final recommendation list and show.

Method in the present invention is not only applied to the retrieval of picture and recommends, also can be used for retrieval and the recommendation of various multimedia messagess, for example, on e-commerce website or TV shopping platform, a lot of existing picture description information of commodity, also there is attribute keyword message, such as suitcase commodity, the picture of suitcase both can be provided, also can provide and comprise suitcase brand, type, size etc. keyword message.So information recommendation system range of application of the present invention is very wide, in the present embodiment, do not represent for example the restriction to range of application of the present invention.

The present invention mainly comprises off-line and online two parts, and wherein step S101 and step S102 are off-line part, and step S103 and step S104 are online part.The commending system of e-commerce website (picture is commodity picture) of take is example, off-line part wherein, it forms maximally related other commodity of each commodity for setting up a machine learning model and picture material and key word effectively being combined and calculate sequence, this model can be realized the renewal (describing later) of increment type simultaneously, thereby effectively the new commodity of fast processing, have good extensibility; Online part be wherein according to user to the browsing or commercial product recommending list that the feature such as buying behavior combines different commodity of commodity, thereby provide the recommendation list of a unified dependent merchandise.

Information recommendation system of the present invention, its framework as shown in Figure 2, online part is to user, to provide retrieval and recommendation service on image retrieval platform, searcher is mainly that (picture library refers to the database that stores mass picture information to picture library, for example, for e-commerce website, it has the commodity picture library that comprises magnanimity commodity picture, picture herein not only comprises the picture material of itself, also comprise the keyword message associated with it) retrieve, the Image ID that recording user was retrieved, then from similar kilsyth basalt, inquire relevant picture and form original recommendation list.Recommendation service is mainly comprised of 3 parts: original recommendation list, filtration and sequence, recommendation service first according to user browse or original recommendation list is obtained in buying behavior, system is filtered according to user's preference or merchandise news afterwards, such as commodity automatic fitration undercarriage or that cannot buy; Because user may browse a plurality of commodity, therefore can obtain a plurality of commercial product recommending lists, therefore system also needs the commodity to repeating in these lists to filter, and final system sorts and forms final recommendation list according to their weight the list after filtering.

The object of off-line part is to calculate picture similarity to obtain similar kilsyth basalt.The present invention utilizes sparse linear model effectively image content information and keyword message to be combined and calculate a unified picture analogies degree.

In step S101, from picture, extract two parts: keyword message and image content information, take commodity picture as example, keyword message is mainly the attribute information of commodity, for example brand, type, size, price etc., and image content information is mainly the natural characteristics such as color in image, texture.

Keyword message can naturally be expressed as into a vector space model, and it is all a sparse matrix conventionally, line display attribute information, and each commodity object is shown in list.

And image content information must be converted into a vector space model by associated picture Processing Algorithm, such as by methods such as color histogram, wavelet transformation, the conversion of yardstick invariant features, the color of image, texture, edge angle dot information etc. being analyzed, each pictures is all converted to a vector on feature space, and image content information is not often sparse, for this reason, the present invention can utilize relevant algorithm to convert image content information to sparse matrix, conventionally according to the feature of image, image is converted in vectorial process, the vectorial degree of rarefication that different algorithms produces is inconsistent, and the degree of rarefication of vector has a certain impact to the efficiency of algorithm and accuracy meeting, the image vector degree of rarefication that the present invention adopts CEDD algorithm to produce can satisfy the demands substantially, if need further to adjust degree of rarefication, also can adopt the mode of sparse coding (conventional image statistics method) to adjust.The object of doing is like this mainly to accelerate to calculate the efficiency of picture similarity.By iterative process effectively, the good basic learning in picture material and two different data sources of key word of this model energy is to the similarity of picture, therefore greatly improve the accuracy rate that similarity is calculated, thereby can for example in commodity storehouse, recommend more exactly maximally related commodity to user at the picture library of magnanimity.

For keyword message, the process of its extraction can adopt existing keyword message extractive technique, and the present invention repeats no more.For image content information, its leaching process as shown in Figure 3, comprising:

S201, by picture according to preassigned quantity piecemeal;

S202, each piecemeal is processed and obtained the histogram that comprises 24 minizones by color cell, each minizone represents a kind of color;

The textural characteristics at S203, use MPEG-7 abstract image edge;

S204, to the color characteristic extracting with textural characteristics quantizes and Regularization, obtain the CEDD histogram of every pictures.

CEDD histogram (Color and Edge Directivity Descriptornorm) is current popular abstract image color characteristic and textural characteristics model, extract every pictures characteristic information and only account for 54 bytes of storage space, and compare the models such as MPEG-7, computing cost is smaller, therefore be widely used in image information retrieval field, and obtain good retrieval effectiveness.CEDD histogram is comprised of texture cell and color cell, and texture cell comprises 6 texture regions, and color cell comprises 24 color regions, therefore altogether comprises 6 * 24 regions.

In step S102, thereby can utilize Sparse model to process calculating similarity to keyword message matrix and image content information matrix.

At the conventional model of Collaborative Filtering Recommendation System, comprise neighbourhood model and matrix decomposition model.When neighbourhood model is recommended, computing cost is little, speed is fast, and its effect is based upon to be calculated on article similarity basis exactly; When matrix decomposition model is recommended, effect is better than neighbourhood model, but computing cost is larger, cannot recommend in real time.Therefore neighbourhood model is widely used for recommending in real time occasion, recommends when responding user's new behavior.

The method of calculating article similarity adopts the equidistant tolerance formula of cosine similarity conventionally, is based upon on vector space model basis, based on the separate hypothesis of attribute, is therefore difficult to calculate exactly article similarity; When the method can not be effectively in conjunction with two kinds of item information data sources in addition, therefore can not bring into play several data source information supplementary advantage mutually.

The sparse linear model that the present embodiment adopts can address the above problem effectively, by in conjunction with keyword message and image content information, adopt machine learning algorithm modeling, therefore compare the method for calculating article similarity based on data mapping, sparse linear model can calculate for example similarity between picture of article more exactly, can meet the requirement of real-time recommendation simultaneously.

In the present embodiment, the algorithmic formula of Sparse model is as follows:

$\begin{array}{cc}\underset{\mathrm{<figure-callout\; id="1"\; label="S"\; filenames="HDA00003858225300041.png"\; state="\{\{state\}\}">S</figure-callout>}}{\mathrm{Minimize}}& \frac{1}{2}{\left|\right|M-\mathrm{MS}\left|\right|}_F^2+\frac{\mathrm{\&alpha;}}{2}\left|\right|{F-\mathrm{FS}\left|\right|}_F^2+\frac{\mathrm{\&beta;}}{2}{\left|\right|S\left|\right|}_F^2+\mathrm{\&lambda;}{\left|\right|S\left|\right|}_1,\end{array}$ $s.t\begin{array}{c}S\&GreaterEqual;0,\\ \mathrm{diag}\left(S\right)=0.\end{array};$

Wherein, M, F represent respectively keyword message matrix (m _ij) _{m * n}with image content information matrix (f _ij) _{p * n}, S represents similarity matrix between calculative picture, || || _f, || || ₁represent respectively Frobenius norm and 1-norm, m, p represent respectively key characteristics quantity and the image content features quantity of all pictures, n represents picture number, wherein, the factor of influence of α representative image content information matrix, for regulating the percentage contribution of image content information when calculating picture similarity, β, λ are regularization parameter, for preventing this Sparse model over-fitting.

Keyword message matrix M element value 0 or 1, and image content information matrix F element value is the integer between 0～7, therefore need to be normalized image content information matrix F.The present embodiment is adopted with the following method the every row of F matrix is normalized respectively.

$F(i,j)=\frac{F(i,j)-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}}{\mathrm{Max}\left\{F(i,\&CenterDot;)\right\}-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}};$

Capable, the j column element of F (i, j) representing matrix F i wherein; Min{F (i) } minimum value of the capable all elements of representing matrix F i; Max{F (i) } maximal value of the capable all elements of representing matrix F i, 0≤i<p, 0≤j<n.

In addition, in step S102, for the new picture adding, in order to calculate the similarity between this picture and other pictures, the present embodiment adopts the method for sparse linear model incremental update to complete similarity computation process, this process does not need again all to calculate the similarity between all commodity, therefore can rapidly, effectively carry out model modification, thereby can recommend new picture.The computing formula of sparse linear model incremental update is as follows:

$\begin{array}{cc}\mathrm{<figure-callout\; id="1"\; label="Minmize"\; filenames="HDA00003858225300041.png"\; state="\{\{state\}\}">Minmize</figure-callout>}& \frac{1}{2}{\left|\right|M(\&CenterDot;,n)-\mathrm{MS}(\&CenterDot;,n)\left|\right|}_2^2+\frac{\mathrm{\&alpha;}}{2}\left|\right|{F(\&CenterDot;,n)-\mathrm{FS}(\&CenterDot;,n)\left|\right|}_2^2+\frac{\mathrm{\&beta;}}{2}{\left|\right|S(\&CenterDot;,n)\left|\right|}_2^2+\mathrm{\&lambda;}{\left|\right|S(\&CenterDot;,n)\left|\right|}_1\end{array}$

wherein, M (, n), F (, n) be keyword feature vector and the image content features vector that new picture is corresponding; S (, n) be the similarity between new picture and other pictures; || || ₂represent 2-norm.

In last online part, user can retrieve by system, for example, while retrieving a Target Photo, obtain the ID of the Target Photo of user search, according to the ID of Target Photo, from similar kilsyth basalt, inquire the similar pictures relevant to Target Photo and form original recommendation list.

After obtaining original recommendation list, need arrange original recommendation list, thereby its arrangement comprises, the picture in original recommendation list is filtered and the processing of sorting obtains final recommendation list.

Simultaneity factor also can sort to original recommendation list according to the weight of picture to the list after filtering, and described weight refers to the preference degree of user to picture, and for example, for commodity picture, this weight has reflected the preference degree of user to corresponding commodity.According to user, the preference degree of picture is obtained to final recommendation list according to sorting successively from high to low.Weighted value herein can be the similarity value that previous calculations goes out, and also can adopt existing method to calculate the preference degree of user to each picture, and for example counting user calculates weight to the number of clicks of picture, click frequency, browsing time etc. information.

In addition, because the picture of user search may have a plurality of, the original recommendation list obtaining so also may have a plurality of, now can the picture in all original recommendation list be filtered, and then the picture after filtering being sorted to combine obtains a final recommendation list.For example extract preceding several pictures that sort in each original recommendation list, be combined into a final recommendation list, also can determine according to the weight (weight herein also can be obtained according to aforementioned similar approach) of each Target Photo of retrieval and obtain a final recommendation list by the picture number of required extraction in each original recommendation list.

The present invention of take is applied to e-commerce website as example, illustrate, suppose that original article number is n, while adding a new commodity, M, F, s-matrix become respectively the matrix of m * (n+1), p * (n+1), (n+1) * (n+1), be that matrix M, F add respectively keyword feature vector and the image content features vector that new commodity is corresponding, form matrix M, F n+1 row; Matrix S is added a line one row, i.e. similarity between new commodity and other commodity; By according to above-mentioned computing formula, solve can calculate S (, n), the similarity between new commodity and other commodity namely, while adding keyword feature vector that new commodity is corresponding and image content features vector due to matrix M, F in addition, very little on similarity impact between original article, therefore can with calculated old matrix S (, k), 0≤k<n, and do not need to recalculate S (, k), 0≤k<n, thereby completed the incremental update of sparse linear model, saved a large amount of computing times.

Below by specific embodiment, implementation result of the present invention is described.

Data Collection: experiment adopts female to wrap keyword message and the image content information of collecting 440 female's bags in data ， Taobao.Keyword message comprises the descriptive labelling information such as color, brand, sex, style, size; Image content information is the picture that female wraps.

Grading collection: for each female's bag, 20 female's bags similar or that picture material is similar described in screening key word, allows three people evaluate, and provides the scoring that each female's bag and other female wrap similarity, averages as final scoring.Scoring level 0-5 divides, and 0 expression is least similar, and 5 expressions are the most similar.Screening average mark >=3 are as test set, for assessment of the effect of algorithm.

Evaluation index: the evaluation index that commending system is conventional comprises prec@n, recall@n, MAP, NDCG, MRR, AUC.

Prec@n: while producing front n commercial product recommending for assessment of commending system, the dependent merchandise returning is shared ratio in recommendation list, i.e. accuracy rate.

Recall@n: while producing front n commercial product recommending for assessment of commending system, the dependent merchandise returning is shared ratio, i.e. recall rate in whole dependent merchandises.

MAP: return to the sequencing information of considering commercial product recommending list dependent merchandise when the corresponding real user of the recommended project likes that the Average Accuracy of the rank of preference calculates this index for assessment of commending system, therefore in recommendation list, the sequence of dependent merchandise is more forward, and MAP just may be higher.

NDCG: produce the matching degree between commercial product recommending list and the list of test set reality for assessment of commending system, this index has been considered the sequencing information of recommendation list commodity.The main thought of this index is that commodity that user likes are come before recommendation list than coming and can increase to a greater extent user below and experience, so the higher commodity of the correlativity NDCG value that sorts when forward is higher, otherwise NDCG value is lower.

MRR: be that the accuracy as it reciprocal is got in the sequence that model answer is provided in result in the system of being evaluated, more all problems are averaged, all users are averaged.

AUC: represent ROC (receiver operator curve) area under a curve, it is weighed a commending system commodity that can to what extent user be liked and the commodity of not liking and distinguishes.

Above-mentioned evaluation index is the conventional mechanism of searching algorithm of evaluating, and concrete ins and outs the present invention repeats no more.

During assessment commending system, prec@n, recall@n, MAP, NDCG, MRR, the desired values such as AUC are higher, show that the recommendation results of this recommend method generation is more accurate, have recommendation effect preferably.

On above-mentioned data set, test, respectively These parameters is assessed, to evaluate the effect of sparse linear model.

From Fig. 4 and Fig. 5, can see, in all 6 conventional evaluation indexes, adopt model of the present invention (SpareLinearMode) obviously all good than using separately model (Texual) or the model based on picture material (Visual) based on key word, particularly in this index of MRR, amplification approaches 100%, wherein also can see that the model based on key word is generally good than the model based on picture material, but incorporating image content information is significantly improved to effect after in model of the present invention, this has also verified that model of the present invention combines this two kinds of methods well, thereby can to user, recommend relevant commodity better.

Based on said method, the present invention also provides the information recommendation system of a kind of combining image content and key word, as shown in Figure 6, comprising:

Matrix conversion module 100, for extract picture library picture keyword message and comprise color characteristic and the image content information of textural characteristics, keyword message and image content information are expressed as to vector space model, obtain corresponding keyword message matrix and image content information matrix;

Similarity calculation module 200, for utilizing Sparse model to process keyword message matrix and image content information matrix, the similarity in calculating chart valut between picture obtains similar kilsyth basalt;

Original recommendation list acquisition module 300, for inquiring about the similar pictures of described Target Photo and form original recommendation list from similar kilsyth basalt according to the Target Photo of user search;

Final recommendation list acquisition module 400, obtains final recommendation list and shows for arranging original recommendation list.About ins and outs existing detailed description in detail in method above of above-mentioned module, therefore repeat no more.

Further, as shown in Figure 7, described matrix conversion module 100 comprises:

Minute module unit 110, for by picture according to preassigned quantity piecemeal;

Color processing unit 120, for each piecemeal is processed and obtained the histogram that comprises several minizones by color cell, each minizone represents a kind of color;

Texture processing unit 130, for the textural characteristics at abstract image edge;

CEDD histogram acquiring unit 140, for to the color characteristic extracting with textural characteristics quantizes and Regularization, obtains the CEDD histogram of every pictures.About ins and outs existing detailed description in detail in method above of above-mentioned modular unit, therefore repeat no more.

Further, described similarity calculation module comprises:

Algorithm unit, for according to the similarity between keyword message matrix and image content information matrix computations picture, wherein algorithmic formula is as follows:

$\begin{array}{cc}\underset{\mathrm{<figure-callout\; id="1"\; label="S"\; filenames="HDA00003858225300041.png"\; state="\{\{state\}\}">S</figure-callout>}}{\mathrm{Minimize}}& \frac{1}{2}{\left|\right|M-\mathrm{MS}\left|\right|}_F^2+\frac{\mathrm{\&alpha;}}{2}\left|\right|{F-\mathrm{FS}\left|\right|}_F^2+\frac{\mathrm{\&beta;}}{2}{\left|\right|S\left|\right|}_F^2+\mathrm{\&lambda;}{\left|\right|S\left|\right|}_1,\end{array}$ $s.t\begin{array}{c}S\&GreaterEqual;0,\\ \mathrm{diag}\left(S\right)=0.\end{array};$

Wherein, M, F represent respectively keyword message matrix (m _ij) _{m * n}with image content information matrix (f _ij) _{p * n}, S represents similarity matrix between calculative picture, || || _f, || || ₁represent respectively Frobenius norm and 1-norm, m, p represent respectively key characteristics quantity and the image content features quantity of all pictures, and n represents picture number.

Further, described similarity calculation module comprises:

Normalized unit, for matrix F is normalized:

$F(i,j)=\frac{F(i,j)-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}}{\mathrm{Max}\left\{F(i,\&CenterDot;)\right\}-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}};$

Wherein, capable, the j column element of F (i, j) representing matrix F i; Min{F (i) } minimum value of the capable all elements of representing matrix F i; Max{F (i) } maximal value of the capable all elements of representing matrix F i, 0≤i<p, 0≤j<n.About ins and outs existing detailed description in detail in method above of above-mentioned modular unit, therefore repeat no more.

Further, similarity calculation module also comprises:

Incremental computations unit, for when having new picture to be added into picture library, adopts the computing formula of sparse linear model incremental update to calculate the similarity between other pictures in new picture and picture library, upgrades sparse linear model.

In sum, the present invention calculates the similarity between image with Sparse models coupling keyword message and image content information, comprising with CEDD model, picture material being carried out to feature extraction.Strong adaptability of the present invention, the recommendation request that meets user that can be real-time, simultaneously will be generally higher than being used single model in the effect of recommending, and wherein the amplification in this index of MRR can approach 100%.

Should be understood that, application of the present invention is not limited to above-mentioned giving an example, and for those of ordinary skills, can be improved according to the above description or convert, and all these improvement and conversion all should belong to the protection domain of claims of the present invention.

Claims (13)

1. an information recommendation method for combining image content and key word, is characterized in that, comprises step:

A, extract picture library picture keyword message and comprise color characteristic and the image content information of textural characteristics, keyword message and image content information are expressed as to vector space model, obtain corresponding keyword message matrix and image content information matrix;

B, utilize Sparse model to process keyword message matrix and image content information matrix, the similarity in calculating chart valut between picture obtains similar kilsyth basalt;

C, according to the Target Photo of user search, from similar kilsyth basalt, inquire about the similar pictures of described Target Photo and form original recommendation list;

D, arrange original recommendation list and obtain final recommendation list and show.

2. the information recommendation method of combining image content according to claim 1 and key word, is characterized in that, in described steps A, the process of extracting the image content information that comprises color characteristic and textural characteristics specifically comprises:

By picture according to preassigned quantity piecemeal;

Each piecemeal is processed and obtained the histogram that comprises several minizones by color cell, and each minizone represents a kind of color;

The textural characteristics at abstract image edge;

To the color characteristic extracting with textural characteristics quantizes and Regularization, obtain the CEDD histogram of every pictures.

3. the information recommendation method of combining image content according to claim 1 and key word, is characterized in that, the algorithmic formula of the sparse model of described step B neutral line is as follows:

$\underset{S}{\mathrm{Minimize}}\frac{1}{2}{\left|\right|M-\mathrm{MS}\left|\right|}_F^2+\frac{\mathrm{\&alpha;}}{2}{\left|\right|F-\mathrm{FS}\left|\right|}_F^2+\frac{\mathrm{\&beta;}}{2}{\left|\right|S\left|\right|}_F^2+\mathrm{\&lambda;}{\left|\right|S\left|\right|}_1,$ $s\&CenterDot;t\begin{array}{c}S\&GreaterEqual;0,\\ \mathrm{diag}\left(S\right)=0\&CenterDot;\end{array};$

Wherein, M, F represent respectively keyword message matrix (m _ij) _{m * n}with image content information matrix (f _ij) _{p * n}, S represents similarity matrix between calculative picture, ‖ ‖ _f, ‖ ‖ ₁represent respectively Frobenius norm and 1-norm, m, p represent respectively key characteristics quantity and the image content features quantity of all pictures, and n represents picture number, the factor of influence of α representative image content information matrix, and β, λ are regularization parameter.

4. the information recommendation method of combining image content according to claim 3 and key word, is characterized in that, F is normalized:

$F(i,j)=\frac{F(i,j)-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}}{\mathrm{Max}\left\{F(i,\&CenterDot;)\right\}-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}};$

Wherein, capable, the j column element of F (i, j) representing matrix F i; Min{F (i) } minimum value of the capable all elements of representing matrix F i; Max{F (i) } maximal value of the capable all elements of representing matrix F i, 0≤i<p, 0≤j<n.

5. the information recommendation method of combining image content according to claim 1 and key word, is characterized in that, described step B also comprises:

When having new picture to be added into picture library, adopt the computing formula of sparse linear model incremental update to calculate the similarity between other pictures in new picture and picture library, upgrade sparse linear model.

6. the information recommendation method of combining image content according to claim 1 and key word, is characterized in that, the step that the original recommendation list of described arrangement obtains final recommendation list displaying further comprises:

When obtaining a plurality of original recommendation list, the picture in all original recommendation list is filtered, and the picture after filtering is sorted and obtains final recommendation list.

7. the information recommendation method of combining image content according to claim 1 and key word, is characterized in that, the step that the original recommendation list of described arrangement obtains final recommendation list displaying specifically comprises:

Picture in original recommendation list is filtered and sorted obtain final recommendation list, show described final recommendation list.

8. the information recommendation method of combining image content according to claim 1 and key word, it is characterized in that, the described step of inquiring about the similar pictures of described Target Photo according to the Target Photo of user search and forming original recommendation list from similar kilsyth basalt specifically comprises:

When user search picture, obtain the ID of the Target Photo of user search, according to the ID of Target Photo, from similar kilsyth basalt, inquire the similar pictures relevant to Target Photo and form original recommendation list.

9. an information recommendation system for combining image content and key word, is characterized in that, comprising:

Matrix conversion module, for extract picture library picture keyword message and comprise color characteristic and the image content information of textural characteristics, keyword message and image content information are expressed as to vector space model, obtain corresponding keyword message matrix and image content information matrix;

Similarity calculation module, for utilizing Sparse model to process keyword message matrix and image content information matrix, the similarity in calculating chart valut between picture obtains similar kilsyth basalt;

Original recommendation list acquisition module, for inquiring about the similar pictures of described Target Photo and form original recommendation list from similar kilsyth basalt according to the Target Photo of user search;

Final recommendation list acquisition module, obtains final recommendation list and shows for arranging original recommendation list.

10. the information recommendation system of combining image content according to claim 9 and key word, is characterized in that, described matrix conversion module comprises:

Minute module unit, for by picture according to preassigned quantity piecemeal;

Color processing unit, for each piecemeal is processed and obtained the histogram that comprises several minizones by color cell, each minizone represents a kind of color;

Texture processing unit, for the textural characteristics at abstract image edge;

CEDD histogram acquiring unit, for to the color characteristic extracting with textural characteristics quantizes and Regularization, obtains the CEDD histogram of every pictures.

The information recommendation system of 11. combining image contents according to claim 9 and key word, is characterized in that, described similarity calculation module comprises:

Algorithm unit, for according to the similarity between keyword message matrix and image content information matrix computations picture, wherein algorithmic formula is as follows:

$\underset{S}{\mathrm{Minimize}}\frac{1}{2}{\left|\right|M-\mathrm{MS}\left|\right|}_F^2+\frac{\mathrm{\&alpha;}}{2}{\left|\right|F-\mathrm{FS}\left|\right|}_F^2+\frac{\mathrm{\&beta;}}{2}{\left|\right|S\left|\right|}_F^2+\mathrm{\&lambda;}{\left|\right|S\left|\right|}_1,$ $s.t\begin{array}{c}S\&GreaterEqual;0,\\ \mathrm{diag}\left(S\right)=0\&CenterDot;\end{array};$

Wherein, M, F represent respectively keyword message matrix (m _ij) _{m * n}with image content information matrix (f _ij) _{p * n}, S represents similarity matrix between calculative picture, ‖ ‖ _f, ‖ ‖ ₁represent respectively Frobenius norm and 1-norm, m, p represent respectively key characteristics quantity and the image content features quantity of all pictures, and n represents picture number, the factor of influence of α representative image content information matrix, and β, λ are regularization parameter.

The information recommendation system of 12. combining image contents according to claim 11 and key word, is characterized in that, described similarity calculation module comprises:

Normalized unit, for matrix F is normalized:

$F(i,j)=\frac{F(i,j)-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}}{\mathrm{Max}\left\{F(i,\&CenterDot;)\right\}-\mathrm{Min}\left\{F(i,\&CenterDot;)\right\}};$

Wherein, capable, the j column element of F (i, j) representing matrix F i; Min{F (i) } minimum value of the capable all elements of representing matrix F i; Max{F (i) } maximal value of the capable all elements of representing matrix F i, 0≤i<p, 0≤j<n.

The information recommendation system of 13. combining image contents according to claim 9 and key word, is characterized in that, similarity calculation module also comprises:

Incremental computations unit, for when having new picture to be added into picture library, adopts the computing formula of sparse linear model incremental update to calculate the similarity between other pictures in new picture and picture library, upgrades sparse linear model.

Images