CN104200240B - A kind of Sketch Searching method based on content-adaptive Hash coding - Google Patents
A kind of Sketch Searching method based on content-adaptive Hash coding Download PDFInfo
- Publication number
- CN104200240B CN104200240B CN201410493545.0A CN201410493545A CN104200240B CN 104200240 B CN104200240 B CN 104200240B CN 201410493545 A CN201410493545 A CN 201410493545A CN 104200240 B CN104200240 B CN 104200240B
- Authority
- CN
- China
- Prior art keywords
- sketch
- feature
- window
- app
- algorithm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 94
- 230000000007 visual effect Effects 0.000 claims abstract description 25
- 238000000605 extraction Methods 0.000 claims abstract description 24
- 238000001514 detection method Methods 0.000 claims abstract description 9
- 239000013598 vector Substances 0.000 claims description 28
- 238000004364 calculation method Methods 0.000 claims description 18
- 230000008569 process Effects 0.000 claims description 17
- 238000012545 processing Methods 0.000 claims description 5
- 230000003044 adaptive effect Effects 0.000 claims description 4
- 238000010606 normalization Methods 0.000 claims description 4
- 238000005070 sampling Methods 0.000 claims description 4
- 238000010586 diagram Methods 0.000 abstract description 10
- 230000035945 sensitivity Effects 0.000 abstract 1
- 238000011160 research Methods 0.000 description 10
- 230000011218 segmentation Effects 0.000 description 7
- 238000005516 engineering process Methods 0.000 description 6
- 238000010845 search algorithm Methods 0.000 description 5
- 238000002474 experimental method Methods 0.000 description 4
- 230000003993 interaction Effects 0.000 description 3
- 239000003086 colorant Substances 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 230000004438 eyesight Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 241000282414 Homo sapiens Species 0.000 description 1
- 239000000654 additive Substances 0.000 description 1
- 230000000996 additive effect Effects 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 230000019771 cognition Effects 0.000 description 1
- 230000001149 cognitive effect Effects 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009792 diffusion process Methods 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000012163 sequencing technique Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 230000016776 visual perception Effects 0.000 description 1
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a kind of Sketch Searching method based on content-adaptive Hash coding, it is characterised in that including step in detail below：Constrained based on appearance constraints and diversity, sketch or profile diagram to being retrieved choose candidate window according to its content-adaptive for feature extraction, realize that the information that whole image is included is evenly distributed in each window；Conspicuousness detection method based on key point detects the conspicuousness of characteristic window；Hash algorithm based on local sensitivity combines the local visual feature of sketch or profile diagram, conspicuousness and structure space feature, is compiled into feature Hash codes；The feature Hash codes of sketch or profile diagram are indexed, the similarity between sketch is measured by calculating the Hamming distance between feature Hash codes, and by similarity it is high return to user.Can have precision higher, wider adaptability and stronger matching capacity using the present invention.
Description
Technical Field
The invention relates to the field of image processing, in particular to a sketch retrieval method based on content adaptive hash coding.
Background
In recent years, searching using sketches (e.g., visual material information such as hand-drawn figures, pictures, and three-dimensional models) as input has been a research focus in the field of computer vision. This is because, with the increasing popularity of touch screen devices, people prefer to use gestures, touch pens, and other methods to complete input and interaction of various information with a computer, and such methods can better express the intention of a user and are very simple to operate. Meanwhile, various retrieval tasks are performed in a hand-drawing sketch mode, so that strong convenience is provided for users using handheld input devices (such as Apple iPhone/iPad, Microsoft Surface and other various tablet computers).
In various sketch interaction tasks, the matching problem of different visual information carriers such as sketches, hand-drawn graphs, pictures, 3D models and the like plays an important role in the whole sketch interaction computing field, and the matching problem is also a basic research problem of a sketch-based retrieval method. In order to obtain the result of the perception and the visual perception of the same person as much as possible in the retrieval task, an efficient algorithm is needed to accurately and quickly measure the similarity between the input sketch and the retrieved information, and the research of the retrieval method based on the sketch aims to solve the problem. From the practical value, on one hand, the retrieval method based on the sketch is closely related to the application of the practical engineering technology, and not only has strong field applicability, but also is a plurality of main core technologies based on the sketch application. On the other hand, it also has a strong scientific research value, which explores the fundamental method of computer cognition to create visual patterns.
In a conventional search task, a sketch is usually regarded as a set of a series of hand-drawn strokes, so as to represent information such as a skeleton and an outline of an object. However, other appearance details possessed by the object, such as color, texture, and the like, are inevitably lost in the process of being converted into a sketch. Due to this characteristic, the sketch-based retrieval method is very different from the conventional picture-based retrieval algorithm. For some time recently, research efforts in this area have been trying to answer a question: how can feature information contained in each sketch be reasonably extracted and compared?
Recently, segmentation-based methods are widely used, and such algorithms have proven to be very effective in the field of sketch retrieval and recognition. They typically reduce computational complexity by segmenting the strokes contained in the sketch during implementation. And then, extracting the attributes of topology, geometric shape and the like of the divided strokes. However, accurate stroke segmentation for sketches is difficult to do, especially for pictures or 3D models that contain natural landscapes. This is because the contour features of such visual information usually contain much noise, and perfect stroke segmentation is an almost impossible task to accomplish. Thus, it is this shortcoming that greatly limits the applicability of one class of methods based on stroke segmentation to be used for sketch-based retrieval of pictures or 3D models.
Other research works are based on the traditional picture search algorithm, and search tasks are completed by comparing the similarity of sketches through regional features (patch). First, a sketch is divided equally into small regions, from which different visual feature descriptors are extracted. In general, such methods cover a whole sketch with a grid with overlapping regions or dense windows of equal size, so as to describe the feature distribution of the sketch. This is very suitable for common pictures, because a picture often contains abundant detailed features such as color or texture, but is not suitable for sparse outline images such as sketches containing only limited strokes. In this case, most of the information contained in a sketch will be contained in a few very salient image areas, leaving the remaining areas almost empty. Due to the extreme imbalance, many invalid feature descriptors are generated, which greatly reduce the effectiveness of the calculation after participating in the feature similarity calculation, and moreover, because the descriptors need to be binarized before indexing them by using the hash algorithm, the situation is further worsened because only visual information is further lost.
Another problem of the search method based on the regional characteristics is that the calculation method of the similarity between the sketches is weak and not strictly accurate. A hand-drawn sketch usually contains various line strokes instead of colors or textures to represent different objects, so that the sketch is greatly different from a traditional picture in two aspects: large intra-class differences (because each sketch renderer has more or less his own subjective understanding of the same east and west) and small inter-class differences (due to the lack of important visual information such as color or texture). This results in even a sketch of the same subject containing a large number of dissimilar regional features. Therefore, "similar pictures often contain a large amount of similar regional features", this concept widely used in picture retrieval algorithms is not accurate for sketch-based retrieval methods, and its definition constraint on similarity is too strict and thus becomes inefficient.
Therefore, in view of the above analysis, it can be seen that the existing sketch-based retrieval technology is not perfect, and needs to be improved and developed. The invention researches a retrieval method based on regional characteristics, because the method has higher applicability compared with a method based on stroke segmentation, and intensively solves two problems in the retrieval method based on regional characteristics.
Disclosure of Invention
The invention aims to provide a sketch retrieval method based on content self-adaptive Hash coding, and aims to solve the problems that the existing sketch retrieval method based on Hash coding does not take account of the content distribution characteristics of a sketch when extracting visual features, the adopted method can greatly reduce the effectiveness of calculation, and the calculation method for the similarity between sketches is weak and not strict and accurate.
The technical scheme of the invention is as follows: a sketch retrieval method based on content adaptive hash coding comprises the following specific steps:
step A: based on appearance constraint and diversity constraint, selecting candidate windows for feature extraction in a self-adaptive manner according to the content of the retrieved sketch or outline map, and realizing that the information contained in the whole image is uniformly distributed in each window;
and B: detecting the significance of the characteristic window based on a significance detection method of the key points;
and C: based on a local sensitive hash algorithm, combining local visual features, saliency and structural space features of a sketch or a contour map to compile a feature hash code;
step D: indexing the characteristic hash codes of the sketch or the outline graph, measuring the similarity between the sketch by calculating the Hamming distance between the characteristic hash codes, and returning the result with high similarity to the user.
The sketch retrieval method can also be used for retrieving the picture and the 3D model, and the picture and the 3D model are preprocessed and converted into the contour line images before being retrieved.
The sketch retrieval method comprises the following steps of converting the picture and the 3D model into the contour line image: calculating a salient outline image of the picture by combining an edge extraction algorithm and a saliency detection algorithm of the picture; for the 3D model, a contour projection graph corresponding to the model is calculated according to a matching algorithm based on a visual angle.
The sketch retrieval method comprises a candidate window selection algorithm for feature extraction: firstly, setting a grid of initialized n x n for an input sketch; uniformly sampling m × m initial seeds from the grid; HOG features h are then computed for the windows in each gridi(ii) a Finally, global HOG characteristics are calculated
The sketch retrieval method is characterized in that the appearance constraint is marked as CappSpecifically, it is represented as: capp(h):＝Fapp(h)≥kapp×Fapp(H)
Wherein, thereinFappIs an objective equation of appearance constraint, when FappWhen the calculated value is higher, it indicates that the feature window contains more visual feature information.
The sketch retrieval method, wherein the diversity constraint is marked as CvarSpecifically, it is represented as: cvar(h):＝Fvar(h)≤kvar×Fvar(H)
Wherein, thereinFvarIs a diversity-constrained objective equation, if FvarHas a low value of and FappHigher indicates that the value of each dimension in the window feature vector h is higher.
The sketch retrieval method comprises the following specific steps of detecting the significance of a feature window by a key point-based significance detection method: first, using a Harris-Laplace detector as a sketch saliency extraction tool, for each feature window wiDefinition of its significance kiComprises the following steps:
wherein, Number (S)i) Is shown in the characteristic window wiThe number of the significant points extracted by using a Harris-Laplace detector is used; area (w)i) Is wiIncluding the number of pixels.
The sketch retrieval method comprises the following steps of: let fiFor the purpose of drawing from the feature window wiThe extracted feature vector is first binarized into vectorThen, following the calculation process of similar hash algorithm, corresponding to each windowAnd kiCalculating the value to obtain a characteristic hash code of the window; then, dividing the sketch into two parts in the horizontal direction and the vertical direction respectively to obtain four separated space positions, and performing hash coding on candidate windows positioned on each space position respectively; and finally, sequentially splicing the hash codes on the four spatial positions end to obtain the characteristic hash code representing the whole sketch.
In the sketch retrieval method, the feature descriptor can also be a scale-invariant feature descriptor or a local linear Gabor feature descriptor.
According to the sketch retrieval method, HOG characteristics without normalization processing are selected to describe visual information contained in each sub-window.
The invention has the beneficial effects that: according to the method, the content distribution condition of the characteristics of the sketch or the contour map is taken into consideration in the process of extracting the characteristics of the sketch or the contour map, so that all the characteristics are uniformly distributed in the candidate window as much as possible, and the extracted characteristic vector can better represent the characteristics of a single sketch. Compared with a retrieval algorithm based on stroke segmentation, the characteristics are less influenced by factors such as noise, texture and the like contained in pictures and 3D model contour diagrams; compared with other retrieval methods based on Hash, the algorithm provided by the invention has higher precision, wider adaptability and stronger matching capability. The application of the method can be popularized to other research fields which take Hash coding on the characteristic information as a key technology, and the method has general significance.
Drawings
FIG. 1 is a schematic representation of the results of a search over a variety of data sets using the present invention.
Fig. 2a, 2b, 2c are schematic diagrams of sketch feature window selection strategies.
Fig. 3a, 3b, and 3c are schematic diagrams of calculation results of sketch saliency windows.
FIG. 4 is a flow chart of a sketch region feature hashing algorithm.
Fig. 5 is a schematic diagram of the intermediate result of calculating the picture contour image.
Fig. 6a, 6b are graphs comparing the performance of different components of the method.
Fig. 7a, 7b are graphs comparing performance for searches using different algorithms.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer and clearer, the present invention is further described in detail below with reference to the accompanying drawings and examples.
The sketch-based retrieval algorithm provided by the invention mainly comprises the following three components: combining the self characteristics of the input sketch, and selecting a feature extraction window in a self-adaptive manner according to two types of constraint conditions; adaptively detecting the significance of each characteristic window by using the key point information contained in the window; the two types of feature information are combined with the structural information of the sketch, and the two types of feature information are coded into a hash code through an LSH algorithm to create a feature index. The present invention will be described hereinafter in the following sections of this specification as well.
As shown in fig. 2a, given a sketch, it is first split into a uniform grid of n x n. Then, as shown in fig. 2b, m × m points are uniformly selected at the intersection points of these grids as initial seeds for generating all feature windows, and then, a set Δ w (x, y, i) of sub-windows is defined for the seed points at the position of the sketch (x, y), where Δ w (x, y, i) represents all the ith circles of sub-windows around the seeds. As shown in fig. 2c, where the black dots represent the selected seed point, the 4 nearest sub-windows surrounding the seed point are the 1 st circle of sub-windows x surrounding the seed point, which is denoted as Δ w (x, y, 1); all the sub-windows surrounding and closest to the 1 st circle of sub-windows are the 2 nd circle of sub-windows surrounding the seed pointDenoted Δ w (x, y, 2).
To finally generate a window w (x, y) of the appropriate size for each initial seed, Δ w (x, y, i) needs to be added to w (x, y) iteratively in turn until w (x, y) satisfies a certain conditional constraint indicating that it contains sufficient characteristic information, or when w (x, y) becomes an illegal window, e.g., the window has overflowed the edge of the entire sketch or it becomes too large (greater than one-fourth of the entire sketch).
Therefore, the selection and design of conditional constraints become especially important, which is directly related to the quality of the finally selected feature window, and if the constraints are too strict, most of the windows contain insufficient feature information; on the contrary, the feature window becomes too large, so that a large number of regions are repeatedly contained in the windows, so that the feature information becomes too redundant, and the calculation amount of feature similarity comparison in the next step is increased. The present invention proposes the following two effective constraint strategies.
Because the HOG (Histogram of Oriented Gradients) features of an image have been widely used in the field of computer vision research such as object detection and image retrieval, and have achieved a good effect, the method chooses the HOG features without normalization processing to describe the visual information contained in each sub-window. In the standard HOG feature calculation process, normalization processing is often adopted to remove the influence of light and shadow on the image, and sketches are both black-white two-color images formed by strokes and backgrounds and have no illumination transformation, so that the use of unnormalized HOG features does not influence the accuracy of feature extraction, and the calculation speed is accelerated to a certain extent due to the reduction of calculated amount. Notation vector h ═ b1,b2,…,bnThe HOG features extracted from the window w (x, y) are used as the HOG features, and the delta h (x, y, i) is the HOG feature vector of the window set corresponding to the delta w (x, y, i); note H as the HOG characteristic H vector of the whole sketch, because the histogram is additive, when calculating the HOG characteristic vector H of all sub-windows in the sketchiThe HOG histogram feature H of the whole graph can then be obtained directly by adding up these values, which is then defined asThe appearance constraint of the sketch feature window is defined as follows, denoted CappIt is defined as follows:
Capp(h):＝Fapp(h)≥kapp×Fapp(H) the system of equation 1
Wherein, thereinFappIs an objective equation of appearance constraint, which essentially calculates the mean of the HOG features h. When F is presentappA higher calculated value indicates that the feature window contains more visual feature information, whereas a lower FappThe value indicates that the entire window is almost empty. Obviously, the constraint CappEnsures that each feature window needs to be congestedAnd the visual characteristic information is abundant enough.
The second constraint proposed by the method is called diversity constraint and is denoted as CvarIt is defined as follows:
Cvar(h):＝Fvar(h)≤kvar×Fvar(H) the system of equation 2
Wherein, thereinFvarIs a diversity-constrained objective equation, which is actually the variance of the HOG features h. If FvarHas a low value of and FappA higher value indicates a higher value for each dimension in the window feature vector h. Thus, satisfy CvarThe constrained window will contain more diverse features than a single direction line or segment, etc., which has proven to be very useful in sketch retrieval problems.
Notably, the parameter kappAnd kvarThe influence of the sketch global HOG characteristic H on two constraints is controlled. Through specific experiments, k is found to beappAnd kvarSetting to 0.8 and 1, respectively, will enable best search performance. After the above two types of conditional constraints are defined, taking the position of each initial seed as a starting point, iteratively adding surrounding sub-windows into the initial seed until each window meets the two types of conditional constraints, thereby completing the selection of the feature candidate window. The algorithm following the window selection strategy is recorded as a window conditional constraint selection algorithm, and the work flow of the whole algorithm is summarized as follows:
step a: initializing a grid of n x n for the input sketch;
step b: uniformly sampling m × m initial seeds from the grid;
step c: computing HOG features h for windows in each gridi；
Step d: the global HOG feature H is calculated.
The specific algorithm is as follows:
since it was mentioned above that "two conditional constraints" are to be used "iteratively" to generate candidate windows, a simple and intuitive implementation example is provided here in pseudo code to help the user understand how to use two conditional constraints.
Notably, constraint CappTarget equation FappIs incrementally computable, that is to say it satisfies the equation:
Fapp(h+Δh)＝Fapp(h)+Fapp(Δ h) · (3) · equation 3
As the window grows continuously, only F of the feature vector corresponding to the growing part of the sub-windows is neededappAdding values to the results of the previous iteration without having to recalculate F over a new window resulting from each iterationappThe value of (c). Thus, it is verified during each iteration whether the feature window satisfies the constraint CappIs very rapid. Although FvarCannot be incrementally calculated, but until each feature candidate window is satisfying constraint CappIt does not have to be calculated before, so in summary, the entire characteristic window of the method is calculated relatively quickly.
In experiments, it can be seen that, no matter a scale-invariant Feature descriptor, a Local linear-based Feature descriptor (GALIF) or the aforementioned HOG Feature descriptor is selected, the Feature window selection strategy proposed above in the present invention can significantly improve the retrieval accuracy.
After the selection of the sketch feature windows is completed, the method also provides a method for detecting the significance of each window in the input sketch. For sketch, to characterize its local features, it is more effective to use feature point-based saliency extraction algorithms, such as multi-scale gaussian model, Hessian algorithm, and Harris-Laplace detector, than to use region-based saliency extraction algorithms. This is due to the fact that a sketch typically contains many separate lines and dots, unlike a picture that contains continuous regions. The feature point-based saliency extraction algorithm is designed to find salient points in an image, and is therefore very suitable for detecting nodes or inflection points in a sketch, which are also the key information contained in the sketch. In the invention, the Harris-Laplace detector is found to have better performance in detecting key points of the sketch through experiments, and therefore, the Harris-Laplace detector is used as the sketch significance extraction method.
For each candidate window W belonging to Wi(where W is the result returned by Algorithm 1, described above, and is the set of all selected feature windows), its significance is defined as kiThe formula is as follows:
wherein, Number (S)i) Is shown in the characteristic window wiThe number of the significant points extracted by using a Harris-Laplace detector is used; area (w)i) Is wiIncluding the number of pixels. Intuitively, it can be seen that: a window is more prominent when it is smaller and contains more salient points.
It should be noted that, in order to prevent the saliency of the feature window from being too sensitive to its size and resolution, the above equation uses the quadratic root term of the area at the pixel level of the window to characterize its size. FIG. 3 illustrates intermediate results of detecting feature window saliency for a sketch, wherein FIG. 3a is an example of an input sketch; the open circles in FIG. 3b represent all the keypoints found in this sketch using the Harris-Laplace detector; fig. 3c illustrates the most significant candidate feature window with the top 3 mutually overlapping areas less than 20% as calculated by equation 4 in a block.
After computing the visual descriptors of the candidate feature windows and their saliency, these contents need to be organically integrated in order to enable these information to be effectively used in similarity comparison and indexing. The similarity measurement algorithm based on the Hash can effectively retrieve a large-scale data set, and simultaneously ensures extremely high calculation speed, so that the method uses the locally sensitive Hash algorithm to encode the extracted features, and the specific process is as follows. In the implementation process of the method, the HOG is used as a window characteristic description algorithm.
Let fiFor the purpose of drawing from the feature window wiThe feature vector extracted in (f) is obtained by extracting fiThe first 40% of the bits with the highest value are set as 1, and the rest are set as-1, i.e. f can be setiBinarized into vectorsIt should be noted that, thanks to the two feature window selection constraints proposed by the present algorithm, it is further effectively ensured that each window can contain enough feature information, so that the loss of information is greatly reduced in the binarization process. Then, the invention refers to the calculation process of similar hash algorithm (Sim-hash), and passes through the corresponding window of each windowAnd kiAnd calculating the value to obtain the characteristic hash code of the window.
In actual calculation, the window significance k is calculatediAsThe weight of (c). A sketch contains spatial information that has proven to be a very useful class of features in the field of sketch retrieval. Therefore, in order to include the spatial relationship of the local features in the sketch into the feature hash code, first, the sketch is divided into two parts in the horizontal and vertical directions, respectively, to obtain four separated spatial positions, as shown in the flow (a) in fig. 4. Then, the candidate windows located at each spatial position are respectively hash-coded according to the above-mentioned method, and the whole process is shown in flows (b) to (e) in fig. 4. And finally, sequentially splicing the hash codes on the four spatial positions end to end, and obtaining the characteristic hash code representing the whole sketch as shown in the flow (f). Given any two sketches, pictures or 3D model projected outline images, the hamming distance (HammingDistance) between their feature hash codes is their similarity.
The following section introduces specific implementation details of the present invention, including pre-processing of pictures and 3D models prior to retrieval, specific setting of parameters, feature data indexing scheme, and the entire query process.
The sketch is usually a black and white image only containing contour lines, the picture often contains rich colors and various textures, and the 3D model is a patch set in a three-dimensional space, and obviously, the three data features are quite different, and they cannot be directly compared and retrieved. Therefore, before the search process starts, in order to enable the user to search the picture and the 3D model by using the sketch as an input, the two types of information need to be preprocessed and converted into the sketch-like outline image.
For a given picture, first, a contour map E of the entire picture is calculated using Canny or other picture edge extraction algorithmscHowever, it is clear that the contour image inevitably contains many false edge lines due to background texture. In order to find the main content (i.e. the portion of the picture desired to be retrieved) expressed by the picture, the saliency detection algorithm of the picture is used to mark the portion of the contentAnd the corresponding saliency map is denoted as S. Then, the method also uses a Maximum Filter (MF) to perform a filtering process on S, so that the significant region of the picture can be slightly enlarged, and the loss of the outer contour of the picture main content due to the segmentation error of the algorithm is avoided. Finally, the salient outline E of the picture can be calculated according to the following formula, and the calculation process is shown in fig. 5.
For a given 3D model, the invention computes the profile projection map corresponding to the model according to the View-based matching (View-based matching) algorithm proposed in the paper "Sketch-based shape Retrieval" by Mathias Eitz et al. In the calculation process, in order to ensure the stability of the projection extraction result, the method adopts the Lloyd relaxation algorithm to uniformly sample about 14 projection visual angles from the corresponding surrounding spherical surface of each 3D model through loop iteration. And, heuristic Contours (Suggestive Contours) are used to extract appropriate contour lines from the model projection graph.
Then, whether the image salient profile graph calculated by the method or the projection profile graph of the 3D model is obtained, the image salient profile graph or the projection profile graph of the 3D model is cut out from the original result by using a minimum square bounding box and is scaled to the resolution of 160 × 160, so that the influence of the image size and deformation on the retrieval result is reduced. Next, according to the calculation process of algorithm 1, the input sketch, picture or model profile is divided into 80 × 80 grids, from which 15 × 15 seeds are uniformly sampled. In each feature sub-window, an unnormalized HOG feature histogram containing 8 directions is computed as h in the constraint, so that each feature vector contains 8 dimensions. All feature windows are then scaled to 16 x 16 blocks of pixels for subsequent visual feature extraction. Finally, since the histogram of the HOG features that have not been normalized before is calculated, after normalizing them, the HOG features can be used to describe the feature vector f for each feature window.
Given any sketch as input, measuring the Hamming distance of the characteristic hash codes between the outline diagrams of all sketches, pictures or models in the database and the outline diagrams, and sequencing from small to large, the sketch, the picture or the 3D model which is most similar to the sketch in the database can be found. Since all hash codes are binarized, the hamming distance between them can be calculated very quickly by simple shift and or operation, and the retrieval speed can be very high even if the feature data is not indexed. In order to further accelerate retrieval performance, the characteristic data is indexed according to the invention combined with a Hamming spatial Hash complex indexing algorithm (Fastsearch in Hamming Space with Multi-Index Hash) proposed by Norouzi et al, so that the method can complete each query request in sub-linear time. The flow of the whole search method of the invention is summarized as follows:
firstly, generating a proper contour map E for all pictures or 3D models in a database; secondly, generating candidate characteristic windows w for all the contour maps E according to a conditional constraint selection algorithm; calculating corresponding HOG characteristics f and significance k for each characteristic window w; then generating a characteristic hash code h for each contour map E according to all HOG characteristics f and significance k in the contour map E; and finally, creating an index I for all the characteristic hash codes h.
Calculating the characteristic hash code h of the input sketch S according to the second step to the fourth step of the retrieval processs(ii) a Calculating the query result R from the index I and returning the result R to the user
In order to support and verify the research method and key technology provided by the invention, the method is compared with other latest and most advanced sketch-based retrieval algorithms respectively on three widely used standard data sets. The Magic Sketch dataset was created by Liang et al, which contained a total of 1100 sketches, each divided into 55 categories according to what was drawn. These sketches are all based on the MPEG-CE1 trademark image database, UKPTO trademark database of the british trademark patent office, refrigerator appliance diagrams and engineering drawings, representative shapes of which are selected and plotted by 10 persons, the data set being used to verify the validity of the various components of the algorithm proposed by the method. The TU Berlin dataset was constructed by Eitz et al, which contained 31 different topics, each with an example sketch and its corresponding 40 test pictures. The data set is used to verify the validity of the method when used in a sketch-based picture retrieval task. The PSB data set is also created by Eitz et al, and contains a sketch that is easily identifiable for each and corresponds to a class of 3D models in a Princeton Shape Benchmark (PSB). Thus, the data set may be used to verify the validity of the method when used in a sketch-based 3D model retrieval task.
In order to evaluate the performance of the sketch-based retrieval technology provided by the invention, the method adopts Recall (Recall), Precision (Precision) and dataset Benchmark Score (Benchmark Score) as performance evaluation indexes. In order to evaluate the existing method more intuitively, the relation between the Recall or Precision and the size of the candidate graph result set is given in the form of a curve (PR curve). As the number of returned results increases, the precision ratio will gradually increase and the precision ratio will gradually decrease, which is determined by the precision ratio and the calculation method of the precision ratio. When the window of the number of the returning structures is increased, the higher the denominator of the precision ratio is, the lower the precision ratio is; for the recall ratio, the larger the window is, the more the number of returned related results is, the larger the numerator of the recall ratio is, and the larger the value of the recall ratio is. Obviously, in the recall ratio graph and the precision ratio graph, the higher the curve is, the better the retrieval effect is, because the higher the curve is, the higher the corresponding recall ratio or precision ratio is in the case of returning the same number of candidate vector graphs; and vice versa. The data set Benchmark score is proposed by Mathias Eitz et al in the paper "Sketch-Based Image Retrieval: Benchmark and Bag-of-FeaturesDescriptors" as another index for evaluating the performance of data Retrieval algorithms. The index judges the quality of the retrieval algorithm by comparing Kendall grade correlation coefficients between ranking results obtained by the retrieval algorithm and artificially labeled real results after given query input under the same retrieval data set. The higher the benchmark score obtained by the search algorithm is, the closer the search result of the algorithm is to the cognitive result of human beings. In addition, the invention tests the time taken by the retrieval algorithm to complete a retrieval task to evaluate the runtime performance of the proposed solution.
Two constraints C for selecting a feature window proposed for verifying the present algorithmapp(formula 1) and Cvar(equation 2) effectiveness, first, an algorithm baseline for feature extraction Using overlapping grids (Grid) is implemented according to the algorithm proposed by Konstantinos Bozas in the paper "Large Scale Sketch Based image retrieval Using Patch Hashing". Then, use it with constraint C onlyappAnd simultaneous use of constraint CappAnd CvarThe present algorithm of (a) was compared. Feature window significance k proposed for validating the present algorithmiValidity of, in the above algorithm implementation, kiAre all set to 1 and all C are used togetherapp、CvarAnd kiThe implementation of (c) was compared. FIG. 6a shows the search performance achieved by the various algorithms on the Magic Sketch dataset as described above, from which it can be seen that the two feature window selection constraints proposed by the present algorithm are complementary, and that C is removedapp、CvarAnd kiAny one of the components will cause the retrieval performance of the algorithm to be reduced. Therefore, C proposed by the algorithmapp、CvarAnd kiIs one-out-of-the-box, each component has its effectiveness.
In addition, in order to further show the performance characteristics of the method, the retrieval algorithm implementation after feature extraction is carried out on a uniform grid by using SIFT, GALIF and HOG feature descriptors is respectively compared with the algorithm implementation of feature extraction on the feature extraction area selected by the algorithm by using the SIFT, GALIF and HOG feature descriptors. The PR curve of fig. 6b shows the comparison result of the above algorithm implemented on the Magic Sketch dataset. From the figure, it can be found that the result of feature extraction on the feature extraction area selected by the algorithm is better than that on the uniform grid no matter what kind of feature descriptors are used, thereby proving the universality of the method for different feature descriptors. It is noted that, thanks to the GALIF feature descriptor, a multi-directional and multi-scale feature sampling strategy is used to replace the traditional histogram strategy for characterizing the visual features, so that it can be found from the figure that the best retrieval performance can be achieved by combining the method and the GALIF feature descriptor. However, to ensure the availability of the algorithm, the HOG feature descriptors are still used to compute the visual features of the sketch in a standard implementation of the method, since computing the GALIF feature descriptors incurs a significant time overhead.
At the end of the experiment, the algorithm was also compared to other leading edge sketch-based retrieval algorithms. On the TUBerlin dataset, the algorithm was compared with a search algorithm using Bag-of-Words (BW), Key Shapes (KW), and minimum hash (Min-hash, MH), and the dataset benchmark scores were used to judge the performance of the search algorithm, with the results shown in table 1. In addition, on the Magic Sketch data set, the algorithm is compared with an algorithm based on biased svm (bsvm), geometric relationships (SR), and geometric Proximity (SP); on the PSB dataset, the present algorithm was compared with a search algorithm based on Diffusion Tensor (DT), Grid SIFT (SIFT-Grid), and GALIF (GALIF-Grid). On both data sets, a standard PR curve was used to evaluate the retrieval performance, with the results shown in fig. 7a and 7b, respectively. The above experimental results show that compared with other retrieval algorithms, the method has higher retrieval precision and better performance. Referring to fig. 1, examples of the retrieval results of the method on different data sets are respectively shown.
TABLE 1 reference scores for different algorithms
The Sketch-based retrieval algorithm provided by the invention takes about 1.87 seconds to complete a Sketch retrieval task on a Magic Sketch standard data set. This result was actually measured on a desktop computer equipped with an Intel 3.39GHz quad CPU, 16GB memory, with the implementation code programmed using MATLAB and without parallel optimization. It is worth noting that the method has higher retrieval precision compared with other algorithms on the premise that the basic real-time requirement can still be met.
It is to be understood that the invention is not limited to the examples described above, but that modifications and variations may be effected thereto by those of ordinary skill in the art in light of the foregoing description, and that all such modifications and variations are intended to be within the scope of the invention as defined by the appended claims.
Claims (9)
1. A sketch retrieval method based on content adaptive hash coding is characterized by comprising the following specific steps:
step A: based on appearance constraint and diversity constraint, selecting candidate windows for feature extraction in a self-adaptive manner according to the content of the retrieved sketch or outline map, and realizing that the information contained in the whole image is uniformly distributed in each window;
and B: detecting the significance of the characteristic window based on a significance detection method of the key points;
and C: based on a local sensitive hash algorithm, combining local visual features, saliency and structural space features of a sketch or a contour map to compile a feature hash code;
step D: indexing the characteristic hash codes of the sketch or the outline graph, measuring the similarity between the sketch by calculating the Hamming distance between the characteristic hash codes, returning the result with high similarity to the user,
in step B, the method for detecting the saliency of the feature window based on the saliency detection method of the key point includes: first, using a Harris-Laplace detector as a sketch saliency extraction tool, for each feature window wiDefinition of its significance kiComprises the following steps:
wherein, Number (S)i) Is shown in the characteristic window wiThe number of the significant points extracted by using a Harris-Laplace detector is used; area (w)i) Is wiIncluding the number of pixels.
2. The sketch retrieval method of claim 1, wherein the pictures and the 3D models are retrieved, and wherein the pictures and the 3D models are preprocessed to convert them into contour images before being retrieved.
3. The sketch retrieval method of claim 2, wherein the method of converting the picture and the 3D model into the outline image is as follows: calculating a salient outline image of the picture by combining an edge extraction algorithm and a saliency detection algorithm of the picture; for the 3D model, a contour projection graph corresponding to the model is calculated according to a matching algorithm based on a visual angle.
4. The sketch retrieval method of claim 1, wherein a candidate window selection algorithm for feature extraction: firstly, the methodSetting a grid of initialization n x n for the input sketch; uniformly sampling m × m initial seeds from the grid; HOG features h are then computed for the windows in each gridi(ii) a Finally, global HOG characteristics are calculated
5. A sketch retrieval method as claimed in claim 1 or 3 wherein said appearance constraint is denoted CappSpecifically, it is represented as: capp(h):＝Fapp(h)≥kapp×Fapp(H)
Wherein,h represents the HOG direction gradient histogram feature vector corresponding to the window, H represents the HOG feature vector corresponding to the whole input sketch, n represents the dimension of the HOG feature vector H, b represents the dimension of the HOG feature vectoriRepresenting the value, k, in the ith dimension of the feature vector happIs a predefined coefficient for controlling the global parameter Fapp(H) Influence of (A) FappIs an objective equation of appearance constraint, when FappWhen the calculated value is higher, it indicates that the feature window contains more visual feature information.
6. The sketch retrieval method as claimed in claim 5, wherein said diversity constraint is denoted as CvarSpecifically, it is represented as: cvar(h):＝Fvar(h)≤kvar×Fvar(H)
Wherein,h represents the HOG direction gradient histogram feature vector corresponding to the window, H represents the HOG feature vector corresponding to the whole input sketch, n represents the dimension of the HOG feature vector H, b represents the dimension of the HOG feature vectoriRepresenting the value, k, in the ith dimension of the feature vector hvarTo be predeterminedCoefficient of sense for controlling the global parameter Fvar(H) Influence of (A) FvarIs a diversity-constrained objective equation, if FvarHas a low value of and FappHigher indicates that the value of each dimension in the window feature vector h is higher.
7. The sketch retrieval method as claimed in claim 1, wherein the method of compiling the characteristic hash code comprises: let fiFor the purpose of drawing from the feature window wiThe extracted feature vector is first binarized into vectorThen, following the calculation process of similar hash algorithm, corresponding to each windowAnd kiCalculating the value to obtain a characteristic hash code of the window; then, dividing the sketch into two parts in the horizontal direction and the vertical direction respectively to obtain four separated space positions, and performing hash coding on candidate windows positioned on each space position respectively; and finally, sequentially splicing the hash codes on the four spatial positions end to obtain the characteristic hash code representing the whole sketch.
8. The sketch retrieval method of claim 4, wherein the feature descriptor is further selected from a scale-invariant feature descriptor or a local linear Gabor feature descriptor.
9. The sketch retrieval method of claim 4, wherein the HOG features without normalization processing are selected to describe the visual information contained in each sub-window.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410493545.0A CN104200240B (en) | 2014-09-24 | 2014-09-24 | A kind of Sketch Searching method based on content-adaptive Hash coding |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410493545.0A CN104200240B (en) | 2014-09-24 | 2014-09-24 | A kind of Sketch Searching method based on content-adaptive Hash coding |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN104200240A CN104200240A (en) | 2014-12-10 |
| CN104200240B true CN104200240B (en) | 2017-07-07 |
Family
ID=52085529
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410493545.0A Active CN104200240B (en) | 2014-09-24 | 2014-09-24 | A kind of Sketch Searching method based on content-adaptive Hash coding |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN104200240B (en) |
Families Citing this family (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105069173B (en) * | 2015-09-10 | 2019-04-19 | 天津中科智能识别产业技术研究院有限公司 | The fast image retrieval method of Hash is kept based on the topology for having supervision |
| CN106126581B (en) * | 2016-06-20 | 2019-07-05 | 复旦大学 | Cartographical sketching image search method based on deep learning |
| CN106126585B (en) * | 2016-06-20 | 2019-11-19 | 北京航空航天大学 | The unmanned plane image search method combined based on quality grading with perceived hash characteristics |
| CN108694406A (en) * | 2017-04-08 | 2018-10-23 | 大连万达集团股份有限公司 | The method compared for the X-Y scheme goodness of fit in engineering |
| CN107402974B (en) * | 2017-07-01 | 2021-01-26 | 南京理工大学 | Sketch retrieval method based on multiple binary HoG descriptors |
| CN108154155A (en) * | 2017-11-13 | 2018-06-12 | 合肥阿巴赛信息科技有限公司 | A kind of jewelry search method and system based on sketch |
| CN108694411B (en) * | 2018-04-03 | 2022-02-25 | 南昌奇眸科技有限公司 | Method for identifying similar images |
| CN108897746B (en) * | 2018-04-03 | 2022-02-08 | 南昌奇眸科技有限公司 | Image retrieval method |
| CN108681555B (en) * | 2018-04-08 | 2019-08-02 | 天津大学 | A kind of sketch image search method returned based on shape |
| CN109033144B (en) * | 2018-06-11 | 2021-10-22 | 厦门大学 | Three-dimensional model retrieval method based on sketch |
| US10248664B1 (en) | 2018-07-02 | 2019-04-02 | Inception Institute Of Artificial Intelligence | Zero-shot sketch-based image retrieval techniques using neural networks for sketch-image recognition and retrieval |
| CN109960738B (en) * | 2019-03-15 | 2020-12-08 | 西安电子科技大学 | Large-scale remote sensing image content retrieval method based on depth countermeasure hash learning |
| CN111528834B (en) * | 2020-03-25 | 2021-09-24 | 西安电子科技大学 | Real-time SAR image target detection system and method |
| CN112115292A (en) * | 2020-09-25 | 2020-12-22 | 海尔优家智能科技（北京）有限公司 | Picture searching method and device, storage medium and electronic device |
| CN113282775B (en) * | 2021-05-27 | 2023-10-03 | 上海焱祺华伟信息系统技术有限公司 | Similar crowd expansion method based on local sensitive hash algorithm |
| CN115114966B (en) * | 2022-08-29 | 2023-04-07 | 苏州魔视智能科技有限公司 | Method, device and equipment for determining operation strategy of model and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN100573523C (en) * | 2006-12-30 | 2009-12-23 | 中国科学院计算技术研究所 | A kind of image inquiry method based on marking area |
| KR100848034B1 (en) * | 2007-03-23 | 2008-07-23 | 한양대학교 산학협력단 | Moment-based local descriptor using scale invariant feature |
| CN101576896A (en) * | 2008-05-09 | 2009-11-11 | 鸿富锦精密工业（深圳）有限公司 | Retrieval system and retrieval method for similar pictures |
-
2014
- 2014-09-24 CN CN201410493545.0A patent/CN104200240B/en active Active
Also Published As
| Publication number | Publication date |
|---|---|
| CN104200240A (en) | 2014-12-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104200240B (en) | A kind of Sketch Searching method based on content-adaptive Hash coding | |
| Li et al. | A comparison of 3D shape retrieval methods based on a large-scale benchmark supporting multimodal queries | |
| US10528620B2 (en) | Color sketch image searching | |
| Cheng et al. | Global contrast based salient region detection | |
| US10489914B2 (en) | Method and apparatus for parsing and processing three-dimensional CAD model | |
| KR20140079262A (en) | Image retrieval method, real-time drawing prompting method and devices thereof | |
| Yang et al. | Object matching with hierarchical skeletons | |
| Ma et al. | Unsupervised multi-class co-segmentation via joint-cut over $ L_ {1} $-manifold hyper-graph of discriminative image regions | |
| CN101996245B (en) | Form feature describing and indexing method of image object | |
| Saavedra et al. | Sketch-based 3D Model Retrieval using Keyshapes for Global and Local Representation. | |
| Seidl et al. | Automated classification of petroglyphs | |
| Deng et al. | Efficient 3D face recognition using local covariance descriptor and Riemannian kernel sparse coding | |
| Szeliski et al. | Feature detection and matching | |
| Liu et al. | A novel rock-mass point cloud registration method based on feature line extraction and feature point matching | |
| Limper et al. | Mesh Saliency Analysis via Local Curvature Entropy. | |
| Zhang et al. | A survey on freehand sketch recognition and retrieval | |
| Wang et al. | An eigen-based motion retrieval method for real-time animation | |
| Zhao et al. | Learning best views of 3D shapes from sketch contour | |
| CN104751463A (en) | Three-dimensional model optimal visual angle selection method based on sketch outline features | |
| JP4570995B2 (en) | MATCHING METHOD, MATCHING DEVICE, AND PROGRAM | |
| Mussarat et al. | Content based image retrieval using combined features of shape, color and relevance feedback | |
| Banerjee et al. | Image retrieval with visually prominent features using fuzzy set theoretic evaluation | |
| CN106384127B (en) | The method and system of comparison point pair and binary descriptor are determined for image characteristic point | |
| JP4324123B2 (en) | Model data display program, model data display device, and model data display method | |
| Wu et al. | Similar image retrieval in large-scale trademark databases based on regional and boundary fusion feature |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |