CN103268363A - Elastic HOG (histograms of oriented gradient) feature-based Chinese calligraphy image retrieval method matched with DDTW (Derivative dynamic time wrapping) - Google Patents

Abstract

The invention discloses an elastic HOG (histograms of oriented gradient) feature-based Chinese calligraphy image retrieval method matched with DDTW (derivative dynamic time wrapping), and belongs to the technical field of information treatment. The method comprises the following steps of: firstly, carrying out pretreatment operation aiming at Chinese calligraphy images, so as to obtain a single character image; dividing an input image into different sizes of grid blocks by adopting an elastic grid technology according to the pixel density distribution of image texts; calculating the HOG features in each grid block; rebuilding EHOG features of the entire character image from the HOG features inside each grid block by an overlapping technology; storing the character image features into a database as character indexing results; extracting the EHOG features of the input character images during retrieving; then carrying out matching search in an indexing database based on a DDTW matching algorithm, and returning the retrieval result on the basis of a specific similarity threshold. The method does not need to utilize an OCR (optical character reader), and has the advantages of high accurate rate, good robustness, simple method, low cost and the like.

Description

A kind of Chinese calligraphy's image search method based on elasticity HOG feature and DDTW coupling

Technical field

The invention belongs to technical field of information processing, relate to the search method of a kind of Chinese calligraphy image, relate in particular to a kind of Chinese calligraphy's image search method based on elasticity HOG feature and DDTW coupling.

Background technology

Chinese calligraphy is a kind of wordsmanship and valuable cultural heritage of having handed down to posterity 3000, mainly is collected in museum and library.Great majority among them all have very high value, but because can not be by random browsing, so can not be known by the public.Such calligraphy document for easy reference, many mechanisms provide the scanning version of these calligraphy samples, as shown in Figure 1.Therefore, index and retrieval technique are indispensable efficiently.Chinese calligraphy is very special, mainly has the characteristic of the following aspects:

1) change: calligraphy is write with writing brush and ink, thereby causes it to have bigger handwriting thickness to change than the word of writing with pen or pencil.Calligraphy has different writing styles in the different dynasties, and the great majority in them have not been used now.

2) degenerate: the calligraphy sample is usually owing to ink fades, paper pollutes and other disadvantageous natural causes are degenerated, to such an extent as to can't identification.

3) deformation: calligraphy has embodied calligraphist's individual character, and the calligraphist is often deliberately with a kind of uncommon mode writing words, as the style of calligraphy characterized by hollow strokes.

For file and picture, a kind of traditional character search method is exactly earlier file and picture to be carried out character recognition, and the result based on identification retrieves then.But for the calligraphy file and picture, because writing of calligraphic character is very random, and may exist various forms of image degradations, so character identification rate is very low, be difficult to make up searching system efficiently based on identification.A kind of feasible method is exactly character picture not to be carried out explicit identification, but directly extracts feature from the calligraphy character, mates based on feature then.This method can be removed loaded down with trivial details and complicated character training and identifying from, makes that the structure of searching system is simple and efficient.Two gordian techniquies of this method are exactly how to extract validity feature to reach the coupling of how to carry out between feature.Existing feature extracting method at handwritten Chinese character retrieval can not well directly apply in the retrieval of handwriting image, and feature matching method generally all adopted the dynamic matching method of DTW, but this method performance is also very limited.

Summary of the invention

At the search problem of Chinese calligraphy's image, the present invention proposes a kind of new for elastic mesh and the synthetic feature extracting method of local histogram of gradients feature, i.e. EHOG feature; Based on the DDTW matching process character feature is carried out online coupling then.

Chinese calligraphy's image search method step based on elasticity HOG feature and DDTW coupling of the present invention is as follows:

(1) at Chinese calligraphy's image, carries out pretreatment operation earlier, to obtain single character picture;

(2) adopt the elastic mesh technology, according to the picture element density distribution of pictograph, input picture is divided into the gridblocks of different sizes;

(3) calculated direction histogram of gradients HOG feature in each gridblock;

(4) the HOG feature in each gridblock is carried out the EHOG feature that has just obtained whole character picture connected in series;

(5) deposit the character picture feature in database as character index result;

(6) when retrieval, to the character picture extraction EHOG feature of input, then based on the DDTW matching algorithm, in the index database, carry out matched and searched, return result for retrieval based on a specific similarity threshold.

The present invention adopts derivative dynamic time warping (DDTW) algorithm to carry out literal location, because DDTW has utilized the shape facility of matching sequence, so it can be applied in the task of calligraphy character seach better, has brought higher retrieval rate.

A distinguishing feature of the present invention is exactly at Chinese calligraphy's image, abandon character recognition technologies (OCR) fully, directly utilize the feature of character picture, innovatively with elastic mesh technology and traditional HOG feature combination, propose to be more suitable for a kind of novel feature--the elasticity local direction histogram of gradients (EHOG) in Chinese calligraphy's Chinese character index, and utilized DDTW matching algorithm relatively more commonly used in the speech recognition to finish the retrieval of literal.The direct search method based on picture material that the present invention proposes need not to utilize OCR, has higher accuracy rate, good robustness, and advantages such as method is simple, with low cost are arranged.

Description of drawings

Fig. 1 is picture format Chinese calligraphy document example;

Fig. 2 is EHOG feature extraction process flow diagram;

Fig. 3 is q (x, bilinear interpolation synoptic diagram y) (12 direction posts) for direction;

Fig. 4 is EHOG feature extracting method example;

Fig. 5 is with " it " the part result for retrieval as polling character.

Embodiment

Embodiment one: the Chinese calligraphy's image search method based on elasticity HOG feature and DDTW coupling of present embodiment, step is as follows:

(1) at Chinese calligraphy's image, removes artificial seal earlier, eliminates pretreatment operation such as noise, Character segmentation, to obtain single character picture.

(2) adopt the elastic mesh technology, according to the picture element density distribution of pictograph, input picture is divided into the gridblocks of different sizes.

(3) definition grid primitive, namely its size is the grid cell less than a gridblock unit, the size of minimum grid primitive is width and highly is 1 pixel.The size of grid primitive can be set up on their own by the user.

(4) to each gridblock unit, the grid that therefrom finds all to satisfy the definition of grid primitive can be overlapping between these grids.As shown in Figure 4, concrete lookup method is: from the upper left corner of grid cell, first along continuous straight runs carries out the slip of individual element to the right, up to the border that arrives grid cell, thereby obtains a series of grid primitive; Get back to the position in the upper left corner then, vertically to pixel of lower slider, and along continuous straight runs carries out the slip of individual element to the right, up to the border that arrives grid cell, thereby gets back a series of grid primitive; According to top step, just stop to slide up to the lower boundary of arrival gridblock and the intersection of right margin.The above-mentioned grid primitive that obtains is arranged according to sequencing, each grid primitive is extracted the HOG feature, then the feature of all grid primitives is carried out the HOG feature that has just constituted gridblock connected in series and describe.

(5) the HOG feature of all gridblocks is carried out connected in series, the feature that has just constituted this character picture is described.

(6) deposit the character picture feature in database as character index result.

(7) when retrieval, to the character picture extraction EHOG feature of input, then based on the DDTW matching algorithm, in the index database, carry out matched and searched, return result for retrieval based on a specific similarity threshold.

(8) basic procedure of DDTW coupling is as follows:

(a) the character picture characteristic sequence of supposition retrieval is Q=q ₁, q ₂..., q _l... q _n, its characteristic sequence length is n, certain the character picture characteristic sequence in the index database is S=s ₁, s ₂, L, s _j, L, s _m, its characteristic sequence length is m.

(b) since the character picture characteristic sequence length of retrieval input not necessarily equate with character picture characteristic length in the index database, can have multiple corresponding relation like this.Suppose that certain bar characteristic of correspondence path is W, then can represent with following formula:

$\left\{\begin{array}{cc}{w}_k={(i,j)}_k& \\ W=w_1,{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="HSA00000911284100011.png,HSA00000911284100013.png"\; state="\{\{state\}\}">w</figure-callout>}}_2,\&CenterDot;\&CenterDot;\&CenterDot;w_k,\&CenterDot;\&CenterDot;\&CenterDot;,w_K& \max (n,m)\&le;K<n+m-1.\end{array}\right.$

(c) for Q and S, the size that we define matching distance is the match is successful whether standard.Matching distance based on DTW is defined as follows:

D(ij)＝d(q _i，s _j)+min{D(i-1，j-1)，D(i-1，j)D(i，j-1)}

Here, d (q _is _j)=(q _i-s _j) ², what i, j represented respectively is certain one dimension of Q and S.

In the present invention, we use d (q ' _i, s _j) replacement d (q _is _j), wherein

Embodiment two: Chinese calligraphy's image is carried out at first need removing pretreatment operation such as artificial seal, image binaryzation, elimination noise, Character segmentation earlier, to obtain single character picture before the index.In the index stage, cut apart good calligraphy character picture to one, at first extract the EHOG feature, then this feature is stored as the index information of this character feature.When actual retrieval, certain the docuterm image to user's input at first extracts the EHOG feature, based on the DDTW matching algorithm all character pictures in the index database is carried out matched and searched then.All images matching similarity in the index database is returned to the user greater than the character picture of certain specific threshold as result for retrieval.Idiographic flow and method that two gordian technique EHOG feature extraction in this flow process and DDTW dynamically mate are as follows:

1、EHOG：

In order to adapt to the characteristics of Chinese calligraphy's literal, the present invention proposes a new feature and describe operator: EHOG, it is the improvement version that the HOG feature is described operator.Fig. 2 has provided the process flow diagram about the EHOG characteristic extraction procedure.G _xAnd G _yRepresent horizontal gradient and VG (vertical gradient) respectively.

As shown in Figure 2 and original HOG be not both, the present invention uses the elastic network(s) technology of formatting that pretreated image is divided into grid heterogeneous.The advantage of elastic mesh is according to stroke intensity the character picture of importing to be divided into empty grid.Owing to there is polytype writing brush word, as the variation of position, size and degree of tilt, therefore than input picture being divided into grid of uniform size, it is more rational feature extracting method that elastic mesh is divided.Like this, two strokes that identical character is identical more may have identical sequential areas, thereby also have similar feature to describe.The division methods of elastic mesh is as follows:

1), to each the some p in the input picture (i, j), calculate current point to the city block distance c of nearest stain (i, j).(i j) is defined as for the weighted point density d of each point

Here i, the horizontal ordinate of j presentation video mid point supposes that picture traverse is I, highly is J, i=1 then, 2 ..., I, j=1,2 ..., J.

2), the dot density with weighting projects to horizontal direction and vertical direction respectively.Level and characteristic projection function are respectively $H\left(i\right)=\underset{j=1}{\overset{J}{\mathrm{\&Sigma;}}}d(i,j)$ With $V\left(j\right)=\underset{i=1}{\overset{I}{\mathrm{\&Sigma;}}}d(i,j).$

3), the coordinate mapping relations are as follows:

$\left\{\begin{array}{c}x\left(i\right)=L\&times;\underset{k=1}{\overset{i}{\mathrm{\&Sigma;}}}H\left(i\right)/\underset{k=1}{\overset{I}{\mathrm{\&Sigma;}}}H\left(i\right)\\ y\left(j\right)=L\&times;\underset{k=1}{\overset{j}{\mathrm{\&Sigma;}}}V\left(j\right)/\underset{k=1}{\overset{J}{\mathrm{\&Sigma;}}}V\left(j\right)\end{array};\right.$

Here L represents the size of the virtual image intending generating, namely longly and wide is L pixel.

Based on above-mentioned conversion, original image just corresponds on the virtual grid image of a L * L, and the division of carrying out grid by a specific unified yardstick on the virtual grid corresponds to the division effect that original image is exactly a non-linear grid.

HOG drops on " number of votes obtained " (quantity) of the pixel on the different directions post, in order to constitute direction histogram according to the Grad of pixel in the statistics grid.Suppose that image I is arranged (x, y), G _xAnd G _yRepresent horizontal gradient and VG (vertical gradient) respectively, then have:

So, we just can obtain pixel (x, Grad m y) and direction θ:

$m(x,y)=\sqrt{{G_x}^2+{G_y}^2}---\left(2\right)$

With

q(x，y)＝R(G _x，G _y) (3)。

In the formula (3), R represents amount of orientation G _x, G _yBetween angle.At last, to obscure situation in order reducing, " number of votes obtained " of pixel to be carried out bilinear interpolation to the adjacent direction post of this point.That is to say that (x is that its nearest both direction post adds m altogether (x y) opens ticket, as shown in Figure 3 y) to each pixel.Wherein, the number of votes obtained of direction post 1 is And the number of votes obtained of direction post 0 is

Here T represents the number of direction post.

The division of direction post can be carried out " no symbol " division in 0 ° to 180 ° interval, or carries out " symbol is arranged " in 0 ° to 360 ° interval and divide.In SIFT and HOG, it is better that " no symbol " divides the experiment effect of dividing than " symbol is arranged ", and this is because the contrast difference of prospect and background does not have differentiation information in its target image.And in experiment of the present invention, use the dividing mode of " symbol is arranged ", because in character picture, prospect and background generally can not be obscured, and have more intense differentiation.Suppose to have us to utilize elastic mesh that input picture is divided into M ' N grid, and T direction post arranged, we just can obtain a histogram that NT direction post of M wound arranged so.

The present invention is unit with the grid primitive, carries out feature extraction, and it is connected in series that all grid primitive features are described operator, describes operator as gridblock HOG feature.The grid primitive is made up of adjacent grid on m ' n the space, so the grid primitive feature is the vector of a m wound nT dimension.These grid primitives use the method (as shown in Figure 4, each grid primitive is made up of 2 * 2 grid lattice, so for each character picture, just can obtain (7-2+1) * (7-2+1) individual grid primitive) of width or the height of an overlapped grid.Therefore, for each character picture, divide altogether (M-m+1)? (Nn+1) individual grid primitive.Finally, obtain a dimension and be (M-m+1)? (Nn+1) the EHOG feature of wound m n T is described operator.

2、DDTW

After the characteristic sequence of handwriting image is described operator extraction by above-mentioned feature and come out, then enter the characteristic matching stage.Fig. 5 has provided a matching result example.Jing Dian matching process is DTW the most, but the present invention has adopted a kind of DDTW of improving one's methods, and experimental result shows that this method is more effective in the handwriting image retrieval.

The ultimate principle of DTW:

Suppose to have two time series Q and S, their length is respectively n and m, has:

Q＝q ₁，q ₂，…，q _i，…q _n (4)

S＝s ₁，s ₂，L，s _j，L，s _m (5)

DTW has constructed the transition matrix of a m ' n, and (i, j) element has comprised some q _iWith a s _jBetween apart from d (q _is _j) (we use Euclidean distance usually here, i.e. d (q _i, s _j)=(q _i-s _j) ²), and represent q _iAnd s _jBetween calibration relation.Alignment path W is made of several continuous elements in the transition matrix, and it has set up mapping relations between sequence Q and S.Alignment path is defined as follows:

$\left\{\begin{array}{cc}{w}_k={(i,j)}_k& \\ W=w_1,{\mathrm{<figure-callout\; id="2"\; label="w"\; filenames="HSA00000911284100011.png,HSA00000911284100013.png"\; state="\{\{state\}\}">w</figure-callout>}}_2,\&CenterDot;\&CenterDot;\&CenterDot;w_k,\&CenterDot;\&CenterDot;\&CenterDot;,w_K& \max (n,m)\&le;K<n+m-1\end{array}---\left(6\right)\right.$

Wherein W is from w ₁=(1,1) beginning is to w _k=(n m) finishes." cost " in this path is as giving a definition:

In other words, the DTW mode that is based on dynamic programming (DP) is determined its best alignment path:

D(i，j)＝d(q _i，s _j)+min{D(i-1，j-1)，D(i-1，j)，D(i，j-1)} (8)

Though DTW is obtaining great success aspect the searching similar sequences, it may produce unexpected result sometimes.In our experiment, DTW mates the character picture of the two dimension on the Y-axle by calibration X-axle (time series).Deviation may appear in DTW in this case: identical word is occurring repeatedly, and certain of the sequence image of one of them word " trough " " trough " more corresponding than another word is low, and the ascending velocity of one of them is littler than another.Suppose that we get two some q among sequence Q and the S _iAnd s _j, their value is identical.But q _iBe to be in an ascendant trend, and s _jBe to be in the downward trend, although we can judge intuitively this moment should be not just these 2 set up mapping relations, think that be corresponding fully at these 2 yet DTW can be wrong.In order to address this problem, we have used improvement algorithm---the DDTW of DTW.DDTW does not directly get the value of sequence as the coupling foundation, but considers the tendency of sequence.By the sequence on the Y-axle being carried out a differentiate, just can obtain the tendency of this sequence.So in DDTW, use d (q ' _I,s _j) replacement d (q _i, s _j), wherein

Like this, calibrating mode just is not simply based on sequential value, but the shape facility of sequence (slope and extreme value).

In order to verify the validity of the inventive method, famous calligraphist's " Zhao Mengpin " calligraphy work is scanned and is picture format, and one has 17 width of cloth calligraphies and paintings.After carrying out Character segmentation, this data set comprises 14,302 words altogether.Experimental result shows that average accuracy rate (mean Average Precision) retrieved in the individual character that the inventive method can reach more than 90% on this data set.Further, the EHOG feature that the present invention proposes is compared traditional HOG feature, retrieves average accuracy rate and can improve more than 8%; Based on the EHOG feature, utilize DDTW to carry out the average accuracy rate of retrieval that matching ratio utilizes DTW to mate and to improve more than 1%.

Claims (5)

1. the Chinese calligraphy's image search method based on elasticity HOG feature and DDTW coupling is characterized in that described method comprises the steps:

(1) at Chinese calligraphy's image, carries out pretreatment operation earlier, to obtain single character picture;

(2) adopt the elastic mesh technology, according to the picture element density distribution of pictograph, input picture is divided into the gridblocks of different sizes;

(3) in each gridblock, calculate the HOG feature;

(4) the HOG feature in each gridblock is carried out connected in series, obtain the EHOG feature of whole character picture;

(5) deposit the character picture feature in database as character index result;

(6) when retrieval, to the character picture extraction EHOG feature of input, then based on the DDTW matching algorithm, in the index database, carry out matched and searched, return result for retrieval based on a specific similarity threshold.

2. the Chinese calligraphy's image search method based on elasticity HOG feature and DDTW coupling according to claim 1 is characterized in that the concrete steps of described step (3) are as follows:

(1) definition grid primitive, namely its size is the grid cell less than a gridblock unit;

(2) to each gridblock unit, therefrom find all to satisfy the grid of grid primitive definition, the grid primitive that obtains is arranged according to sequencing, each grid primitive is extracted the HOG feature, then the feature of all grid primitives is carried out the HOG feature that has just constituted gridblock connected in series and describe.

3. according to the Chinese calligraphy's image search method based on elasticity HOG feature and DDTW coupling according to claim 2, it is characterized in that the described lookup method that satisfies the grid of grid primitive definition is: from the upper left corner of grid cell, elder generation's along continuous straight runs carries out the slip of individual element to the right, up to the border that arrives grid cell, thereby obtain a series of grid primitive; Get back to the position in the upper left corner then, vertically to pixel of lower slider, and along continuous straight runs carries out the slip of individual element to the right, up to the border that arrives grid cell, thereby gets back a series of grid primitive; According to top step, stop to slide up to the lower boundary of arrival gridblock and the intersection of right margin.

4. according to the Chinese calligraphy's image search method based on elasticity HOG feature and DDTW coupling according to claim 1 and 2, it is characterized in that described EHOG Feature Extraction step is: the HOG feature of all gridblocks is carried out connected in series, the EHOG feature that has just constituted this character picture is described.

5. according to the Chinese calligraphy's image search method based on elasticity HOG feature and DDTW coupling according to claim 1, it is characterized in that the basic procedure of described DDTW coupling is as follows:

(a) the character picture characteristic sequence of supposition retrieval is Q=q ₁, q ₂..., q _i... q _n, its characteristic sequence length is n, certain the character picture characteristic sequence in the index database is S=s ₁, s ₂, L, s _j, L, s _m, its characteristic sequence length is m;

(b) suppose that certain bar characteristic of correspondence path is W, then represent with following formula:

$\left\{\begin{array}{cc}{w}_k={(i,j)}_k& \\ W=w_1,w_2,\&CenterDot;\&CenterDot;\&CenterDot;w_k,\&CenterDot;\&CenterDot;\&CenterDot;,w_K& \max (n,m)\&le;K<n+m-1\end{array}\right.;$

(c) for Q and S, whether standard that the size of definition matching distance is that the match is successful is defined as follows based on the matching distance of DTW:

D (i, j)=d (q _i, s _j)+min{D (i-1, j-1), D (i-1, j), D (i, j-1) }, d (q _i, s _j)=(q _i-s _j) ²), what i, j represented respectively is certain one dimension of Q and S;

(d) in DDTW, use d (q ' _i, s _j) replacement d (q _i, s _j), wherein

Images