CN106528826A - Deep learning-based multi-view appearance patent image retrieval method - Google Patents

Abstract

The present invention provides a method for retrieving multi-view appearance patent images based on deep learning. The method includes: preprocessing the appearance patent images, normalizing the appearance patent images in terms of dimensions and scales, and performing multi-view images according to views Distinguish; Construct a multi-view deep convolutional neural network, construct a convolutional neural network for parallel processing of multiple views, and use a fully connected convolutional network after feature fusion according to the spatial position relationship of the views; pre-trained during network training The network parameters are optimized and adjusted; after the training is completed, the image classification and feature extraction are performed in the image library, and the image features are stored in the corresponding category; during image retrieval, similar images and similarity are fed back according to the category first and then the feature . The multi-view deep convolutional neural network of the present invention fuses the correlation between views and improves retrieval accuracy.

Description

A Multi-View Appearance Patent Image Retrieval Method Based on Deep Learning

technical field

The invention relates to the field of image retrieval, in particular to a deep learning-based multi-view appearance patent image retrieval method.

technical background

Under the background of our country's in-depth implementation of the innovation-driven development strategy, our province proposes to accelerate the upgrading of advantageous traditional industries. Patented technology is the engine that guides the development of the technology industry. Patents contain 90%-95% of all scientific and technological information in the world, and the disclosure of technical information is 1 to 2 years earlier than other carriers. Design patents have become an important way to protect the intellectual property rights of enterprises, safeguard their own rights and interests, and protect inventions and creations.

At present, there are two main categories of image retrieval based on appearance patents. The first category is text-based retrieval, which is the most commonly used method. The main defect is that images cannot be marked with appropriate text, which is the so-called A picture is worth a thousand words. The results of such retrievals lead to highly skewed results that cannot be retrieved.

The second method is to use the method of image search. The traditional method is to extract the features of the image, such as shape, texture, color, etc., through the so-called "optimal feature extraction algorithm" such as Gabor filter and SIFT. The distance between features is used for similarity comparison. In these methods, each view of the design patent image is treated as an independent image for feature processing, resulting in low retrieval accuracy.

Design patent images usually use multiple views (such as 4 views or 6 views) to represent the appearance of the invention object. The multi-views of design patent images are an organic whole, therefore, designing the retrieval of design patent images that organically use these views is a problem that needs to be solved in this technical field.

Contents of the invention

Aiming at the deficiencies in the prior art, the present invention provides a deep learning-based multi-view appearance patent image retrieval method, constructs a multi-view convolutional neural network deep learning architecture, and uses pre-trained network parameters as the initial weights of the learning network At the same time, the image is divided into channel features according to the view, pooled and fused according to the spatial position relationship of the view, and subsequent feature extraction and classification are performed. This method greatly improves the accuracy of image retrieval results, and solves the problem of lack of organic fusion of features between multiple views in the process of image retrieval of appearance patents.

According to the design scheme provided by the present invention, a deep learning-based multi-view appearance patent image retrieval method specifically includes the following steps:

Step 1. Preprocessing of the appearance patent image, normalize the scale of the appearance patent image, normalize the image dimension, and distinguish and classify each view of the appearance patent image; divide the appearance patent image data set into a test data set and a training data Set in two parts.

Step 2. Construct a multi-view deep learning network, construct a network containing 3 layers of convolution for each type of view according to the seven-view and seven-way branch, and then use pooling for fusion, then 3-layer full connection, and finally classify through Softmax Output, using the pre-trained network parameters as the initial weights of the network.

Step 3. Image feature extraction and classification, using the training samples to train the multi-view deep convolutional neural network, adjusting the network parameter weights, and obtaining the multi-view deep learning network model after the trained network. The test set and training level images are passed through the network model to calculate the feature representation and classification of the image.

Step 4. The search results are sorted and output by similarity. After the image to be retrieved is pre-processed, the features and categories of the image are extracted through the deep learning network, and the distance is compared with the similar image features. According to the value of the distance Sort the feedback output from small to large, and output the corresponding image.

Beneficial effects of the present invention: the present invention aims at the lack of organic application of multi-dimensional views of patent drawings in the existing appearance patent image retrieval, utilizes deep convolutional neural networks to construct multi-view convolutional neural networks, and performs corresponding views according to the path of views Convolution processing, considering the spatial position relationship of views, performs pooling and fusion processing, which excavates the internal relationship between views and greatly improves the accuracy of image retrieval.

Description of drawings

Fig. 1. Schematic flow chart of the present invention

Fig. 2. The flowchart provided by the embodiment of the present invention.

detailed description

In order to make the purpose, technical solution, and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present invention, not to limit the present invention.

Embodiment 1, as shown in Figure 1, a deep learning-based multi-view appearance patent image retrieval method is characterized in that it includes:

In step 101, the appearance patent image is preprocessed, the appearance patent image scale is normalized, the image dimension is normalized, and each view of the appearance patent image is distinguished and classified; the appearance patent image data set is divided into a test data set and the training data set.

As mentioned above, in step 101, image scale normalization is to adjust the image to the same scale.

Preferably, the scale is 128*128.

As mentioned above, in step 101, image dimension normalization is to convert a two-dimensional grayscale image into a three-dimensional image similar to RGB format.

Preferably, the value of the pixel corresponding to the R, G, and B channels of the newly added image is the same as the value of the pixel corresponding to the grayscale image.

In step 102, construct a multi-view deep learning network, construct a network containing 3 layers of convolution for each type of view according to the seven-view and seven-way branch, and then use pooling for fusion, then 3-layer full connection, and finally through Softmax For classification output, use the pre-trained network parameters as the initial weight of the network.

As mentioned above, in step 102, the seven views and seven branches are the main view, left view, right view, top view, bottom view, rear view and three-dimensional view branches.

As mentioned above, in step 102, the pre-trained network parameters are network parameters trained based on ImageNet.

Preferably, the VGG-M model is selected as the network parameter.

As mentioned above, in step 102, the initial parameters of the seven-way network branches are the same as the network architecture.

As mentioned above, in step 102, the pooling is used for fusion and the maximum value based on pad is used for fusion.

Preferably, the rules of fusion are fused according to the adjacency of the view space positions.

Preferably, the size of the pad is 2*2.

In step 103, the image features are extracted and classified, and the training samples are used to train the multi-view deep learning network, and the network parameter weights are adjusted to obtain the multi-view deep learning network model after the trained network. The test set and training level images are passed through the network model to calculate the feature representation and classification of the image.

As mentioned above, in step 103, the feature of the image is the multi-dimensional image feature xi output after the ReLU before Softmax, and the category Ck of the image is obtained after Softmax.

Further, compression coding is performed on the multi-dimensional image features xi.

In step 104, the search results are sorted and output by similarity. After the image to be retrieved is pre-processed, the features and categories of the image are extracted through the deep learning network, and the distance is compared with the image features of the same type. The value of is sorted from small to large for feedback output, and the corresponding image is output.

As mentioned above, in step 104, the category of the image to be retrieved through the deep learning network is Cn, then only the multi-dimensional image features previously calculated and stored for the image of the Cn category are considered in the subsequent similarity comparison. The distance used may be Euclidean distance, Mahalanobis distance, or the like.

Embodiment 2: As shown in FIG. 2 , a deep learning-based multi-view appearance patent image retrieval method is characterized in that it includes:

In the initial image input, for the network parameter adjustment training process, each input image is a multi-view of multiple appearance patents. When retrieving, at least one image to be retrieved can be input.

In step 201, the image size is normalized, and the input image size is unified to facilitate subsequent feature analysis and extraction.

In step 202, the three channels of RGB are unified, and the image inputted as RGB is not processed; for the grayscale image, a new image is constructed, and the value of the pixel corresponding to the R, G, and B channels of the new image is the same as the gray value of the pixel. The corresponding pixels of the image have the same value.

In step 203, view classification is to input the input images into corresponding channels according to their view labels.

In step 204, CNN1 uses a 3-layer neural convolutional network to extract features of the image. Initially, the CNN1 network parameters of each channel are the same, but after training, the network parameters of each channel may be inconsistent.

In step 205, the multi-view pooling process performs pooling and fusion on each view channel according to its spatial position using a 2*2 Pad maximum value method.

In step 205, CNN2 uses a 3-layer fully connected deep convolution method to extract image features, the activation function of the last layer of CNN2 uses ReLU, and then outputs the high-level features of the image, and associates the feature with the image name for storage .

In step 207, the Softmax classifier classifies the features in step 205 to obtain the category of the image.

In step 208, the category of the image is further associated and stored with the characteristics obtained in step 205. In the subsequent image retrieval, first judge the category of the image, and then compare the distance of the image features under this category, and output the specified number of images from small to large according to the calculated distance.

Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: it can still be Modifications are made to the technical solutions described in the foregoing embodiments, or equivalent replacements are made to some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims (6)

1. a kind of multi views appearance patent image search method based on deep learning, it is characterised in that include：Appearance patent figure As pretreatment, multi views depth convolutional neural networks, network training and image characteristics extraction and classification are built, retrieval result is similar Degree sequence output；

Described appearance patent image pretreatment, the appearance patent image to being input into are normalized and standardization processing；

Described construction multi views depth convolutional neural networks, build multichannel depth according to the maximum view classification of appearance patent Convolutional neural networks, and using pre-training model parameter as initiation parameter；

Described network training and image characteristics extraction and classification, adjust pre-training using using the appearance patent image of labelling Deep learning network parameter.The feature and classification information of appearance patent image collection are extracted, and is stored；

Described retrieval result sequencing of similarity output, image to be retrieved is compared with the characteristics of image in picture library, defeated Go out the high image of similarity and its similarity.

2. the multi views appearance patent image search method based on deep learning according to claim 1, it is characterised in that Described appearance patent image pretreatment is included appearance patent image dimension normalization, image dimension normalization, while will be outer Each view for seeing patent image makes a distinction classification；Appearance patent image data set is divided into into test data set and training data Collection two parts.

3. the multi views appearance patent image search method based on deep learning according to claim 1, it is characterised in that The construction multi views depth convolutional neural networks include branching into each class view construction comprising 3 floor convolution according to seven road views Network, merged using pond afterwards, be then 3 layers it is complete connect, carry out classification output finally by Softmax, profit With the network parameter of pre-training as network initial weight.

4. the multi views appearance patent image search method based on deep learning according to claim 1, it is characterised in that Described network training, image characteristics extraction and classification include carrying out multi views depth convolutional neural networks using training sample Training, is adjusted to network parameter weight, obtains the multi views deep learning network model after training network.By test set and The image of training level is calculated the character representation and its classification of image by network model.

5. the multi views appearance patent image search method based on deep learning according to claim 1, it is characterised in that Retrieval result sequencing of similarity output, including by image to be retrieved after Image semantic classification, through deep learning Network, extracts the feature and classification of image, and the row distance that enters between similar characteristics of image compares, according to the numerical value of distance Sequence feedback output from small to large, and corresponding image is exported.

6. the multi views appearance patent image search method based on deep learning according to claim 1, it is characterised in that Comprise the following steps：

In step 201, picture size normalization, by the picture size unification of input, facilitates follow-up analysis and extraction of features；

In step 202., unified is tri- channels of RGB, and the image to being input into is not processed for the image of RGB；To gray level image, structure Width new images are built, the R of new images, G, the value of channel B respective pixel are identical with gray level image respective pixel value；

In step 203, view classification, the image that will be input into are separately input in corresponding passage according to its view label；

In step 204, CNN1, image is extracted to the feature of image using 3 layers of neural convolutional network.Being initially, The CNN1 network parameters of each paths are identical, according to training after, the network parameter of each paths is likely to occur inconsistent；

In step 205, multi views pondization is processed, and each road view passage is adopted the Pad maximum of 2*2 according to its locus Mode carries out pondization fusion；

In step 205, CNN2, extracts characteristics of image using the depth convolution mode of 3 layers of full connection, last layer of CNN2 Activation primitive adopts ReLU, after which the high-level characteristic of output image, and this feature and image name are associated storage；

In step 207, Softmax graders, the feature of step 205 is classified, and obtains the classification of image；

In a step 208, the further associated storage of characteristic that will be obtained in the classification of image and step 205.In follow-up image During retrieval, first determine whether the classification of image, then, the distance of movement images feature under the category, according to it is calculated away from From exporting the picture number specified from small to large.