CN103530403B - A kind of structurized Image Description Methods - Google Patents

Abstract

The invention belongs to image retrieval technologies field, particularly relate to a kind of structurized Image Description Methods, including: obtain the image being trained, and to 3 layers of tree label of object configurations each in image, form training set；Extracting the low-level feature of each object of image in training set, training obtains the class of all candidates, the grader that subclass and Attribute Relative are answered, and forms the intermediate data required for next step modeling；Structural environment random field models, training obtains model parameter；For image to be described, first carry out image segmentation, be partitioned in image to be described, the object comprised, then extract the low-level feature of each object in image to be described,；Utilize the conditional random field models (CRF) constructed and training to obtain model parameter afterwards, use and use the long-pending belief propagation algorithm of maximum, the tree label of object each in image to be described, is predicted.The present invention can improve the discrimination between image and image, produces and preferably retrieves result.

Description

A kind of structurized Image Description Methods

Art

The invention belongs to image retrieval technologies field, particularly relate to a kind of structurized Image Description Methods.

Background technology

Use more abundant semantic information describe piece image no matter to understand this image or from Web region retrieval This image is all and weight.On the one hand, when in the face of a width new image, people are the most inquisitive is the object in image Belong to which class (being such as a kind of animal or a kind of vehicles), when after the category information getting it, further people Wonder it is belonging to which subclass (belonging to birds, still fall within felid), additionally, every kind of object all to have oneself peculiar Attribute information, ratio is such as whether have feather, if can fly, if food meat etc..By these information, people can be from multiple Angle understands piece image more accurately, gets more knowledge about objects in images simultaneously.On the other hand, at figure As searching field, owing to computer is to use low-level feature to represent piece image, the retrieval be so given by computer is tied Fruit can not well mate the retrieval of user and be intended to, for this " semantic gap " that overcome field of image search to exist, also Us are needed to use more abundant more accurate semantic information to describe piece image.

Emerge various Image Description Methods, such as use single label to retouch State the object in image and belong to animal or plant, although this description method specifies the classification of objects in images, but institute The information contained is very limited, and in order to overcome this shortcoming, people are created that a tag library, then chooses in tag library therewith Relevant multiple labels describe certain object in image, but tag library is the most limited, it is impossible to cover in nature All objects, the most just create picture based on attribute and describe method, and this kind of method uses the attribute information of object, is such as No meeting flies, if having feather etc. to go to describe piece image, even if advantage of this is that and encountering the image that a width was not met, People still can use some basic attribute informations to go to describe it, thus obtains some perceptual knowledge to image.No matter What angle is these Image Description Methods be from, uses what information to describe image, and their target is provided to obtain Semantic information more abundant in image.

Summary of the invention

It is an object of the invention to provide a kind of new structured image and describe method, use the tree of 3 layers Semantic label describes piece image, so that the content describing image is abundanter.

The structurized Image Description Methods that the present invention proposes comprises the steps:

The first step, obtains the image being trained, and to 3 layers of tree label of object configurations each in image, is formed Training set:

(1) obtain the image being trained, build image collection IMG；

(2) use image segmentation algorithm to be partitioned in set IMG the object included in each image, constitute collection of objects OBJ；

(3) being labeled each object in set OBJ, the content of mark includes the class belonging to object, subclass and The attribute having, forms class set CLASS, subclass set SUBCLASS and community set ATTRIBUTE；

(4) according to markup information, each object configurations in set OBJ is included class-subclass-attribute 3 layers tree mark Sign, form the tag set Y corresponding with OBJ, the element one_to_one corresponding of two set；

(5) class set CLASS has the most identical element, each element in sequential scan CLASS, identical element Only retain one, form the class set Cla of candidate, its subclass set SUBCLASS and community set ATTRIBUTE is done identical Scanning, obtain the subclass set Subcl and the community set Attri of candidate of candidate；

(6) structure includes: collection of objects OBJ, tag set Y, the class set Cla of candidate, the subclass set Subcl of candidate And training set Tr of the community set Attri of candidate；

Second step, the low-level feature of each object of the image in extraction training set, training obtains the class of all candidates, son The grader that class and Attribute Relative are answered, forms the intermediate data required for next step modeling, and step is as follows:

(1) local binary patterns (LBP) feature of each object, structural feature set X in collection of objects OBJ is extracted；

(2) according to characteristic set X and class set CLASS, training obtains gathering each element w in Cla_iSVM classifier SVM_w_i, constitute the grader set S corresponding with class_w, in like manner, according to characteristic set X and subclass set SUBCLASS, training Obtain each element v in subclass set Subcl_jSVM classifier SVM_v_j, constitute the grader set S corresponding with subclass_v, According to characteristic set X and community set ATTRIBUTE, training obtains gathering each element u in Attri_kSVM classifier SVM_ u_k, constitute the grader set S answered with Attribute Relative_u；

(3) in training set Tr, grader set S is calculated_uIn each grader SVM_u_kPrecision ratio and recall curve (PR curve), obtains grader SVM_u according to this PR curve_kThreshold value thre_k, constitute and S_uCorresponding threshold value set Threshold；

(4) in training set Tr, add up each element w in the class set Cla of candidate_iSubclass set with candidate Each element v in Subcl_jCo-occurrence probabilities p_ij, i.e. tag set Y has w simultaneously_iAnd v_iPhysical quantities account in OBJ total Object number N_mRatio；Training set Tr is added up each element v in the subclass set Subcl of candidate_iProperty set with candidate Close each element u in Attri_kCo-occurrence probabilities g_jk, i.e. set Y has u simultaneously_kAnd v_iPhysical quantities account for total thing in OBJ Body number N_mRatio, additionally, in training set Tr statistics containing candidate subclass set Subcl in element v_iBut do not contain candidate Community set in element u_kProbability q_jk, i.e. set Y has v_iBut there is no u_kPhysical quantities account for total object number N in OBJ_m Ratio；

(5) structure comprises: grader set S_w, S_v, S_u, threshold value set Threshold, probability statistics p_ij, g_jk, q_jk Intermediate data, in case next step modeling use；

3rd step, structural environment random field models (CRF), training obtains model parameter；

4th step, for image to be described, first carries out image segmentation, is partitioned in image to be described, the object comprised, According still further to the method for aforesaid second step, extract the low-level feature of each object in image to be described,；Afterwards, the 3rd step structure is utilized Conditional random field models (CRF) and the training made obtain model parameter, use and use the long-pending belief propagation algorithm of maximum, to be described, In image, 3 layers of tree label of each object are predicted.

The present invention uses the tree semantic primitive of 3 layers to describe image, and from this semantic primitive, user is not only The class belonging to objects in images and concrete subclass information can be got, and the genus that objects in images has can be got Property information, thus produce more abundant to picture material and describe more accurately, so can improve between image and image Discrimination, convenient eliminate semantic gap when image retrieval, produce and preferably retrieve result, the present invention is also supplied to simultaneously The methods of exhibiting of a kind of picture material more intuitively of user, i.e. carrys out the class of organization charts's picture with 3 layers of tree, subclass, Attribute information, facilitates the understanding image that user is more prone to.

Accompanying drawing explanation

Fig. 1: CRF model structure schematic diagram.

Fig. 2: the present invention use training set in some examples, wherein the image of the first row be from network download from So image, object therein has used rectangle frame to mark.3 layers of tree label that the second each object of behavior is corresponding.

The left figure of Fig. 3 is natural image, wherein needs the object described to use rectangle frame to mark, and right figure is to use this 3 layers of tree semantic label that the invention of bright proposition dopes.

The left figure of Fig. 4 is natural image, wherein needs the object described to use rectangle frame to mark, and right figure is to use this 3 layers of tree semantic label that the invention of bright proposition dopes.

Detailed description of the invention

Here two width images are chosen as image to be described, respectively Fig. 3, the image on the left side in Fig. 4, use in the present invention It is predicted exporting 3 layers of tree semantic primitive by the method described.

The model parameter of condition random field (CRF) is obtained: specifically comprise the following steps that firstly the need of training

1, structure training set step is as follows:

(1) write the image in the retrieval result of crawlers download Google picture searching, constitute image collectionWherein N_dIt it is the total number of images in set IMG；

(2) use image segmentation algorithm to be partitioned in set IMG the object included in each image, constitute collection of objectsWherein N_mIt is the object sum in set OBJ, because piece image there may be multiple thing Body, so N_m≥N_d；

(3) Amazons Mechanical Turk instrument is used to be labeled, each object in set OBJ including object Obj_lAffiliated class class_l, subclass subclass_lAnd the attribute attr having_l1,…,attr_lp, lp represents object Obj_lThe attribute having Number, forms class setSubclass set And community set ${\mathrm{attr}}_{11},\•\•\•,{\mathrm{attr}}_{1p}\•\•\•,{\mathrm{attr}}_{l1},\•\•\•,{\mathrm{attr}}_{\mathrm{lp}},\•\•\•{}_{},{\mathrm{attr}}_{N_{m}1},\•\•\•{\mathrm{attr}}_{N_{m}p},\};$

(4) according to markup information to each object Obj in set OBJ_lConstruct 3 layers of tree label (class-subclass- Attribute) Y_l={ class_l,subclass_l,attr_l1,…,attr_lp, form the tag set corresponding with OBJ(the element one_to_one corresponding of two set)；

(5) class set CLASS has the most identical element, each element in sequential scan CLASS, identical element Only retain one, form the class set of candidateWherein N_wIt is the sum of different elements in set Cla, right Subclass set SUBCLASS and community set ATTRIBUTE does identical scanning, obtains the subclass set of candidateCommunity set with candidateN_vAnd N_uIt is in set Subcl and Attri respectively The sum of different elements；

(6) structure includes: collection of objectsTag setCandidate's Class setThe subclass set of candidateAnd the community set of candidateTraining set Tr.The training set generated is as shown in Figure 2.

2, process data step as follows:

(1) each object Obj in collection of objects OBJ is extracted_l(l=1 ..., N_m) local binary patterns feature (Local Binary Patterns) LBP feature X_l, structureCharacteristic set；

(2) according to characteristic setWith class setTrain Each element w in set Cla_iSVM classifier SVM_w_i, constitute the grader set corresponding with classIn like manner, according to characteristic setWith subclass setTraining obtains gathering each element v in Subcla_jSVM classifier SVM_ v_j, constitute the grader set corresponding with subclassAccording toAnd genus Property setTraining is gathered Each element u in Attri_kSVM classifier SVM_u_k, constitute the grader set answered with Attribute Relative

(3) set of computations S in training set Tr_uIn each grader SVM_u_kPrecision ratio and recall curve (PR is bent Line), obtain SVM_u according to this PR curve_kThreshold value thre_k, constitute and S_uCorresponding threshold value set

(4) each element w in statistics set Cla in training set Tr_i(i=1,2 ..., N_w) and set Subcla In each element v_j(j=1,2 ..., N_v) co-occurrence probabilities p_ij(i=1 ..., N_w, j=1 ..., N_v) (i.e. same in set Y Time there is w_iAnd v_iPhysical quantities account for total object number N in OBJ_mRatio)；In training set Tr in statistics set Subcla often One element v_i(i=1,2 ..., N_v) and gather each element u in Attri_k(k=1,2 ..., N_u) co-occurrence probabilities g_jk (j=1 ..., N_v, k=1 ..., N_u) (i.e. set Y has u simultaneously_kAnd v_iPhysical quantities account for total object number N in OBJ_m's Ratio), additionally, statistics is containing element v in set Subcla in training set Tr_i(i=1,2 ..., N_v) but do not contain set Element u in Attri_k(k=1,2 ..., N_u) probability q_jk(j=1 ..., N_v, k=1 ..., N_u) (i.e. set Y has v_i But there is no u_kPhysical quantities account for total object number N in OBJ_mRatio)；

(5) structure comprises: grader set Threshold value setProbability statistics p_ij(i= 1,...,N_w, j=1 ..., N_v), g_jk(j=1 ..., N_v, k=1 ..., N_u), q_jk(j=1 ..., N_v, k=1 ..., N_u) Intermediate data, in case next step modeling use.

3, build CRF model and training parameter step is as follows:

(1) structure CRF model as shown in Figure 1:

Wherein o₁It is category node, represents the class belonging to object in image, can any one in the class set Cla of candidate Value (i.e. o at individual element₁∈{1,…,N_w})；o₂It is child class node, represents the subclass belonging to object in image, can be candidate Subclass set Subcl in value (i.e. o at any one element₂∈{1,…,N_v})；o_k(k=3 ..., m), m=2+N_uIt is to belong to Property node, represent the attribute that the object in image has, span is 1 and 2 (i.e. o_k∈ 1,2}), wherein 1 represents object not Having this attribute, 2 represent that object has this attribute).So we can use O_l={ o₁,o₂,o₃…,o_mReplace Y_l= {class_l,subclass_l,attr_l1,…,attr_lpRepresent object Obj_lThe 3 layers of tree label having；CRF model Need maximize equation below:

$P\left(O_l\right|X_l)=\frac{1}{Z\left(X_l\right)}\exp \{-E(X_l,O_l)\}---\left(1\right)$

Wherein, X_lRefer to from object Obj_lThe low-level feature extracted, Z (X_l) it is a normalized constant, it is called partition Function, E (X_l,O_l) it is called energy function, it may be seen that maximize formula (1) to be equivalent to minimize energy function, energy letter The formula of number is defined as follows:

Wherein,It is called nodal potential function (node potential), γ₁It is defined as o₁Corresponding graderAt X_lOutput above, γ₂It is defined as o₂Corresponding graderAt X_lOutput above, works as o_k=2 (k= 3 ..., time m), γ_kIt is defined as o_kCorresponding grader SVM_u_k-2At X_lOutput above, works as o_k=1 (k=3 ..., m), γ_k(k =3 ..., m) it is defined as o_kCorresponding grader SVM_u_k-2Threshold value thre_k-2(k-2∈{1,…,N_u})；It is called Limit potential function (edge potential), whereinWork as o_k=1 (k=3 ..., time m),

Work as o_k=2 (k=3 ..., time m),ω={ ω₁,…,ω_m+nBe CRF model needs maximize the parameter that formula (1) obtains；

(2) stochastic gradient descent method (stochastic gradient descent) is used to minimize the energy in formula (2) Flow function E (X_l,O_l), thus obtain the parameter ω={ ω of CRF model₁,…,ω_m+n}。

Secondly, after training obtains CRF model parameter, being next predicted image to be described, step is as follows:

(1) image segmentation algorithm is used to be partitioned into image Image to be described,^tIn the object that comprisesIts Middle N^tRepresenting the object sum being partitioned in image to be described, in Fig. 3, Fig. 4, red frame show use image segmentation algorithm segmentation The object gone out.

(2) Image is extracted^tIn each objectLocal binary patterns feature (Local Binary Patterns) LBP featureL=1 ..., N^t；

(3) model parameter ω={ ω that training obtains is utilized₁,…,ω_m+nMaximization equation below:

Wherein,It is defined asIn first element o₁Corresponding grader?Output above,Definition ForIn second element o₂Corresponding grader?Output above, works as o_k=2 (k=3 ..., time m),Fixed Justice is o_kCorresponding grader SVM_u_k-2?Output above, works as o_k=1 (k=3 ..., m), γ_k(k=3 ..., m) definition For o_kCorresponding grader SVM_u_k-2Threshold value thre_k-2(k-2∈{1,…,N_u})；Node o₁And o₂Between the value on limitNode o₂With node o_k(k=3 ..., m) between the value on limitNeed a point situation discussion, work as o_k=1 (k =3 ..., time m),Work as o_kTime

(4) the long-pending belief propagation algorithm (max-product belief propagation.) of maximum is used to carry out solution formula (3), thus obtain making formula (3) maximized outputThe object i.e. doped's 3 layers of tree label.

3 layers of tree label such as Fig. 3 of final prediction output, in Fig. 4 shown in right side.

Conclusion: the present invention is directed to iamge description problem and propose a kind of structurized description method.From present invention definition 3 In layer tree semantic label, user not only can obtain the class belonging to objects in images, subclass, it is also possible to get object The attribute information having, and class, subclass, the structural relation between attribute, thus produce image more accurate, abundanter Description.

Claims (1)

1. a structurized Image Description Methods, comprises the following steps:

The first step, obtains the image being trained, and to 3 layers of tree label of object configurations each in image, forms training Collection:

(1) obtain the image being trained, build image collection IMG；

(2) use image segmentation algorithm to be partitioned in set IMG the object included in each image, constitute collection of objects OBJ；

(3) being labeled each object in set OBJ, the content of mark includes the class belonging to object, subclass and having Attribute, formed class set CLASS, subclass set SUBCLASS and community set ATTRIBUTE；

(4) according to markup information, each object configurations in set OBJ is included class-subclass-attribute 3 layers tree label, Form the tag set Y corresponding with OBJ, the element one_to_one corresponding of two set；

(5) having the most identical element in class set CLASS, each element in sequential scan CLASS, identical element is only protected Stay one, form the class set Cla of candidate, its subclass set SUBCLASS and community set ATTRIBUTE is done identical sweeping Retouch, obtain the subclass set Subcl and the community set Attri of candidate of candidate；

(6) structure includes: collection of objects OBJ, tag set Y, the class set Cla of candidate, the subclass set Subcl of candidate and Training set Tr of the community set Attri of candidate；

Second step, extracts the low-level feature of each object of image in training set, and training obtains the class of all candidates, subclass with And the grader that Attribute Relative is answered, forming the intermediate data required for next step modeling, step is as follows:

(1) local binary patterns (LBP) feature of each object, structural feature set X in collection of objects OBJ is extracted；

(2) according to characteristic set X and class set CLASS, training obtains gathering each element w in Cla_iSVM classifier SVM_ w_i, constitute the grader set S corresponding with class_w, in like manner, according to characteristic set X and subclass set SUBCLASS, training obtains Each element v in subclass set Subcl_jSVM classifier SVM_v_j, constitute the grader set S corresponding with subclass_v, according to Characteristic set X and community set ATTRIBUTE, training obtains gathering each element u in Attri_kSVM classifier SVM_u_k, structure Become the grader set S answered with Attribute Relative_u；

(3) in training set Tr, grader set S is calculated_uIn each grader SVM_u_kPrecision ratio and recall curve (PR is bent Line), obtain grader SVM_u according to this PR curve_kThreshold value thre_k, constitute and S_uCorresponding threshold value set Threshold；

(4) in training set Tr, add up each element w in the class set Cla of candidate_iWith in the subclass set Subcl of candidate Each element v_jCo-occurrence probabilities p_ij, i.e. tag set Y has w simultaneously_iAnd v_iPhysical quantities account for total object number in OBJ N_mRatio；Training set Tr is added up each element v in the subclass set Subcl of candidate_iCommunity set with candidate Each element u in Attri_kCo-occurrence probabilities g_jk, i.e. set Y has u simultaneously_kAnd v_iPhysical quantities account for total object in OBJ Number N_mRatio, additionally, in training set Tr statistics containing candidate subclass set Subcl in element v_iBut do not contain candidate's Element u in community set_kProbability q_jk, i.e. set Y has v_iBut there is no u_kPhysical quantities account for total object number N in OBJ_m's Ratio；

(5) structure comprises: grader set S_w, S_v, S_u, threshold value set Threshold, probability statistics p_ij, g_jk, q_jkIn Between data, in case next step modeling use；

3rd step, structural environment random field models (CRF), training obtains model parameter；

4th step, for image to be described, first carries out image segmentation, is partitioned in image to be described, the object comprised, then presses According to the method for aforesaid second step, extract the low-level feature of each object in image to be described,；Afterwards, the 3rd step structure is utilized Conditional random field models (CRF) and training obtain model parameter, use and use the long-pending belief propagation algorithm of maximum, to image to be described, In 3 layers of tree label of each object be predicted.