KR20220090367A - Method of searching trademarks and apparatus for searching trademarks - Google Patents

Abstract

A trademark retrieval method includes: receiving a query image by a first learning model executed by one or more processors; extracting, by the first learning model, a first feature vector related to the overall appearance of the query image; calculating, by the one or more processors, a degree of similarity between a second feature vector of a trademark image stored in a database and the first feature vector; receiving, by a second learning model executed by the one or more processors, the query image, wherein the second learning model includes a main model and a first sub-model, wherein the main model divides the query image into N classifications. is configured to classify, the first sub-model is configured to classify the query image into M classifications selected from among the N classifications, and the classification by the main model and the first sub-model is a classification of the same layer; outputting, by the second learning model, at least one classification corresponding to the query image based on the partial image of the query image; selecting at least one image similar to the query image from among trademark images stored in the database based on the similarity between the second feature vector and the first feature vector and the classification of the second learning model; and providing the selected image. The outputting of the at least one classification by the second learning model may include: providing, by the main model, a first classification based on the query image; and providing, by the first sub-model, a second classification based on the query image.

Description

TRADEMARKS AND APPARATUS FOR SEARCHING TRADEMARKS

The present disclosure relates to a trademark search method and apparatus, and more particularly, to a trademark search method and apparatus using two artificial intelligence (AI) models.

In the case of Cypris (kipris.or.kr), which is currently widely used to search for trademarks, through classification criteria such as product classification list, similar group code, trade name, graphic trademark, application number, announcement number, registration number, and applicant You can search for a trademark. You can also search for brand names with similar pronunciations. Also, it is possible to search for a trademark using the Vienna code that classifies the similarity of figures. However, in the case of image retrieval, there is a problem in that images that are not similar are searched even if the Vienna code is the same, or it is difficult to search for images that are similar even though the Vienna code is different. To compensate for this, various methods for classifying trademark images using artificial intelligence have been devised, but it is difficult to find cases that are actually producing results.

Korean Patent Publication No. 10-2017-0083453 Korean Patent Publication No. 10-2019-0030434 Korean Patent Publication No. 10-2019-0098801

The present disclosure provides a trademark search method and apparatus for easily searching for a trademark having a similar figure image by using two models: a model for estimating image similarity and a model for estimating a classification code in a trademark search.

According to one aspect of the present disclosure, a trademark search method using artificial intelligence is disclosed. A trademark retrieval method includes: receiving a query image by a first learning model executed by one or more processors; extracting, by the first learning model, a first feature vector related to the overall appearance of the query image; calculating, by the one or more processors, a degree of similarity between a second feature vector of a trademark image stored in a database and the first feature vector; receiving the query image by a second learning model executed by the one or more processors; outputting, by the second learning model, at least one classification corresponding to the query image based on the partial image of the query image; selecting at least one image similar to the query image from among trademark images stored in the database based on a degree of similarity between the second feature vector and the first feature vector and the at least one classification; and providing the selected image.

In an embodiment, the second learning model includes a main model and a sub-model. The outputting of the at least one classification by the second learning model may include: providing, by the main model, a first classification based on the query image, wherein the first classification is included in the at least one classification; providing, by the sub-model, a second classification based on the query image; and replacing, by the one or more processors, the first classification with the second classification in response to determining that the first classification and the second classification are different.

In an embodiment, the second learning model includes a main model and a sub-model, the main model is configured to classify figures included in the query image into N classifications, and the sub-model divides the query image into the N classifications. It may be configured to classify into M classifications selected from among the N classifications.

In an embodiment, the outputting of the at least one classification by the second learning model comprises: the main model providing a first classification based on the query image, wherein the first classification is included in the at least one classification - to do; determining, by the one or more processors, whether the first classification is included in the M classifications; in response to determining that the first classification is included in the M classifications, providing, by the sub-model, a second classification based on the query image; and replacing, by the one or more processors, the first classification with the second classification in response to determining that the first classification and the second classification are different.

In an embodiment, the second learning model includes a model for performing object detection, and determines a plurality of object likelihood boxes based on a point at which a pixel value changes rapidly in the query image, and an interest of the object likelihood box. It may be configured to perform classification on some images of the query image by calculating a ratio of overlapping regions.

In an embodiment, the selecting of at least one similar image to the query image from among trademark images stored in the database based on the similarity between the second feature vector and the first feature vector and the at least one classification comprises: determining the number of overlapping classifications by comparing one classification with the classifications of the trademark images stored in the database; comparing the at least one classification with the classifications of the trademark images stored in the database and calling a weight according to the number of overlapping classifications and selecting at least one image similar to the query image based on the weight and a degree of similarity between the second feature vector and the first feature vector.

According to one aspect of the present disclosure, a trademark search apparatus is disclosed. The trademark retrieval device includes: a storage configured to store a trademark retrieval model; and a processor configured to execute the trademark search model. the trademark search model includes a first learning model and a second learning model, wherein the first learning model is configured to extract a first feature vector from a query image to obtain a similarity with a second feature vector of a trademark image stored in a database and the second learning model is configured to output at least one classification corresponding to the query image based on the partial image of the query image. The processor is configured to select at least one image similar to the query image from among the trademark images stored in the database based on a degree of similarity between the second feature vector and the first feature vector and the at least one classification.

In an embodiment, the first learning model is configured and trained to measure the similarity of figures, and is trained with a triplet loss function using first learning data classified into N classifications, and the second The learning model is configured and trained to classify figures included in the trademark into the N classifications, and may be learned using the first learning data.

In an embodiment, the second learning model may include a main model and a sub-model. The main model may be configured to classify the query image into N classifications, and the sub-model may be configured to classify the query image into M classifications selected from among the N classifications.

In an embodiment, at least one of the criteria for classifying the at least one classification may be a Vienna code. the main model is configured to provide a first classification based on the query image, the first classification being included in the N classifications; the one or more processors are configured to determine whether the first classification is included in the M classifications; the sub-model is configured to provide a second classification based on the query image in response to determining that the first classification is included in the M classifications; and the one or more processors may be configured to replace the first classification with the second classification in response to determining that the first classification and the second classification are different.

In an embodiment, the second learning model may include a main model and a sub-model. the main model is configured to provide a first classification based on the query image, wherein the first classification is included in the at least one classification, and the sub-model is configured to provide a second classification based on the query image. and the processor may be configured to replace the first classification with the second classification in response to determining that the first classification and the second classification are different.

In an embodiment, the second learning model may include a neural network for performing object detection, and may be configured to extract a plurality of features from the second trademark image to estimate a classification corresponding to each of the plurality of features.

In an embodiment, the processor compares the at least one classification with the classification of the trademark image stored in the database to determine the number of overlapping classifications, and compares the at least one classification with the classification of the trademark image stored in the database to retrieve a weight according to the number of overlapping classifications, and select at least one image similar to the query image based on the weight and the degree of similarity between the second feature vector and the first feature vector.

You can easily search for similar trademark images by simply uploading an image for a trademark search. In addition, it is possible to provide a trademark search engine capable of distinguishing not only the overall external image of a figure but also the similarity of the detailed image of the trademark main part by using the search model using the two artificial intelligence models.

1 is a block diagram of a trademark search environment according to one embodiment.
2 is a block diagram of a terminal according to an embodiment.
3 is a block diagram of a trademark search server according to an embodiment.
4A is a similarity histogram of a similarity image using the first learning model according to an embodiment, and FIG. 4B is a similarity histogram of a random image using the first learning model according to an embodiment.
5A is a similarity histogram of a similarity image using a first learning model and a second learning model according to an embodiment, and FIG. 5B is a similarity histogram of a random image using the first learning model and a second learning model according to an embodiment.
6 is a block diagram illustrating a trademark search model according to an embodiment.
7 is a flowchart illustrating a trademark search method according to an embodiment.
8 is a flowchart illustrating a database manufacturing method according to an exemplary embodiment.
9A to 9C are diagrams illustrating classification according to some images in a trademark image according to an embodiment.

Hereinafter, the present disclosure will be described in detail with reference to the drawings. In describing the present disclosure, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the subject matter of the present disclosure, a detailed description thereof will be omitted. In addition, the following examples may be modified in various other forms, and the scope of the technical spirit of the present disclosure is not limited to the following examples. Rather, these embodiments are provided to more fully and complete the present disclosure, and to fully convey the technical spirit of the present disclosure to those skilled in the art.

It is to be understood that the technology described in the present disclosure is not intended to be limited to specific embodiments, and includes various modifications, equivalents, and/or alternatives of the embodiments of the present disclosure.

In connection with the description of the drawings, like reference numerals may be used for like components.

In the present disclosure, expressions such as “have,” “may have,” “include,” or “may include” indicate the presence of a corresponding characteristic (eg, a numerical value, function, operation, or component such as a part). and does not exclude the presence of additional features. In this disclosure, expressions such as "A or B," "at least one of A and/and B," or "one or more of A or/and B" may include all possible combinations of the items listed together. . For example, "A or B," "at least one of A and B," or "at least one of A or B" means (1) includes at least one A, (2) includes at least one B; Or (3) it may refer to all cases including both at least one A and at least one B.

As used in the present disclosure, expressions such as “first,” “second,” “first,” or “second,” may modify various elements, regardless of order and/or importance, and refer to one element. It is used only to distinguish it from other components, and does not limit the components.

A component (eg, a first component) is "coupled with/to (operatively or communicatively)" to another component (eg, a second component); When referring to "connected to", it will be understood that the certain element may be directly connected to the other element or may be connected through another element (eg, a third element). On the other hand, when it is said that a component (eg, a first component) is "directly connected" or "directly connected" to another component (eg, a second component), the component and the It may be understood that other components (eg, a third component) do not exist between other components.

The expression “configured to (or configured to)” as used in this disclosure, depending on the context, for example, “suitable for,” “having the capacity to” ," "designed to," "adapted to," "made to," or "capable of." The term “configured (or configured to)” may not necessarily mean only “specifically designed to” hardware. Instead, in some circumstances, the expression “a device configured to” may mean that the device is “capable of” with other devices or parts. For example, the phrase "a processor configured (or configured to perform) A, B, and C," a module configured (or configured to perform) A, B, and C," refers to a dedicated processor (eg, : embedded processor) or a generic-purpose processor (eg, CPU or application processor) capable of performing corresponding operations by executing one or more software programs stored in a memory device.

Trademark Image Search Using Artificial Intelligence

In the present disclosure, using artificial intelligence means that a learning model including an artificial neural network (ANN) is trained through a large amount of learning data to optimize parameters inside the artificial neural network, and the learned learning model is used. can mean In an embodiment of the present disclosure, it collects and processes image data of trademarks provided by the Korean Intellectual Property Office to create learning data and create a database for storing them. Learning data, trademark data, feature vectors of trademark data, classification of trademark data, etc. may be stored in the database.

The training data may be stored in a separate database. By inputting the trademark images stored in the database to the learning model trained using the learning data, extracting a feature vector from each trademark image and storing it, the image of the trademark to be searched (query image) is When the learning model is input, the similarity is calculated by extracting a feature vector from the query image and comparing it with the stored feature vector. In addition, in the present disclosure, by using artificial intelligence, not only the overall similarity of figures included in the trademark image, but also the degree of similarity of a part of the figure may be determined and utilized. For example, the whole and/or part of a figure is classified by using the learning model, and a figure classification result value is obtained according to the classification of the figure. Then, the final similarity may be calculated by multiplying the similarity of the feature vector and the result of the figure classification.

artificial neural network

The artificial neural network may refer to a computing system based on a biological neural membrane. The artificial neural network may be composed of a plurality of neural network layers. Each neural network layer has a plurality of parameters, and a neural network operation is performed through an operation between the operation result of the previous layer and the plurality of parameters. The parameters of the plurality of neural network layers may be optimized by the learning result of the artificial neural network. The artificial neural network model according to the present disclosure is a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), There is at least one or a combination of these, such as Deep Belief Network (DBN), Bidirectional Recurrent Deep Neural Network (BRDNN), or Deep Q-Networks, but in the above example not limited

In one embodiment, a model using a convolutional neural network is used as the artificial neural network. For example, a specialized convolutional neural network is used to measure the similarity of the overall appearance of an image.

convolutional neural network

Since convolutional neural networks are commonly used for image recognition, we briefly describe them. Convolutional neural networks are mainly used for image recognition because they can maintain spatial information of images. In general, a convolutional neural network may be manufactured by combining a convolutional layer, an activation function, a pooling layer, and the like. The convolution layer is a layer that outputs a feature map by applying a filter (or kernel) expressed as a matrix to input data. Filters can be automatically generated by learning, and optimizing the parameters inside the artificial neural network by training the learning model means optimizing the filter.

A learning model including a convolutional neural network repeatedly extracts and pools a feature map using a plurality of filters from an image, transforms the finally obtained feature map into a fully connected layer, and through this, a feature vector to get

The learning model used in an embodiment of the present disclosure is inception, VGG. Any one of models such as LeNet, AlexNet, ZFNet, VGG, GoogLeNet, ResNet, MobileNet, and EfficientNet may be used or partially modified, but the present invention is not limited thereto.

database creation

The database was prepared as follows.

In the present disclosure, trademark data were provided from Cypris Plus (plus.kipris.or.kr). Cypris Plus provides information on patents, trademarks, and designs that are open to the Korean Intellectual Property Office in bulk data and/or open API methods. At the time of database creation, about 900,000 data were obtained from Cypris Plus. That is, a total of about 900,000 graphic trademarks and composite trademarks of figures and characters were obtained. Duplicate trademarks have been removed here. In addition, since one of the several purposes of the present disclosure is to easily search for similarity of figures, only trademarks classified as image (figure) trademarks are extracted by separating characters from composite trademarks.

In one embodiment, the removal of duplicate trademarks proceeds by parsing the patent applicants from data obtained from Cypris. Through this, the first trademark data (duplicated data) is obtained.

In one embodiment, the step of classifying a trademark to be used as a figure image in the first trademark data is as follows. In order to classify a figure image from the first trademark data, a conventionally known Binary Classification model (hereinafter, a figure extraction model) is used.

See, for example, inception, VGG. Any one of the models such as LeNet, AlexNet, ZFNet, VGG, GoogLeNet, ResNet, MobileNet, and EfficientNet can be used or some modified ones can be used. In one embodiment, at least one of OpenCV and PIL's Image module and a convolutional neural network are used to distinguish figures and characters from a composite trademark. In an embodiment, trademark data may be arbitrarily divided or an area estimated to contain characters may be trained with a convolutional neural network, and figures and characters may be distinguished using the convolutional neural network.

In one embodiment, in the trademark data, the criterion for distinguishing between a character and a trademark is when the figure occupies a certain size or more of the entire trademark, it is classified as a figure image. For example, the predetermined size may be about 50% of the total size. The predetermined size may be changed. In one embodiment, the size ratio may use the actual area of the figure and character, or may use a bounding box. When a character has a design, if the character is included in the figure, it is classified as a figure. As the training data of the figure extraction model, 2000 were randomly selected. That is, a figure extraction model is trained with 2000 trademark images, and the first trademark data is input to the figure extraction model to obtain second trademark data to be used as a figure image.

The figure extraction model was trained using the learning data, and the first trademark data was input to the figure extraction model to obtain the second trademark data (data classified as duplicates and figures) and stored in the database. That is, duplicate trademarks were removed from the trademark data obtained from Cypris Plus, and second trademark data, which is a trademark classified as a figure, was obtained.

Learning data production

In an embodiment, the second trademark data may be used as it is without further classification using classification according to the Vienna code. The Vienna code is a classification code that expresses the figure within a trademark with six digits. The first two digits correspond to the major classification, the middle two digits correspond to the intermediate classification, and the last two digits correspond to the subclassification.

According to an embodiment, the second trademark data may be newly classified and used based on the Vienna code. An additional classification was performed on the classification according to the Vienna code. Accordingly, the second trademark data was classified into N classifications.

In more detail, the second trademark data is classified using the Vienna code. In some cases, classification was performed using all major classifications, medium classifications, and subclassifications and additional classification was performed. For example, if the number of figure images classified according to the Vienna code is sufficient and they are classified into a similar classification, additional classification is not performed. That is, in this case, classification according to the Vienna code is used as it is. If the figure images classified through detailed classification were classified as dissimilar classifications, they were divided into other classifications. That is, the same sub-category but different classifications. The two groups were merged when the Vienna code classifications, which lacked the number of images, were judged to be similar to each other. For example, if the Vienna code is 08-07-03 (macaroni, other pasta), 11-01-06 (chopsticks), and 11-03-07 (bowl), the above three classifications have different Vienna codes and Although the quantity is insufficient, “container” and “noodle food” are commonly included in the picture, so it is judged to be a similar classification, and therefore it has been designated/merged into the category “noodle dish”. In this way, a classification table of 241 classifications was prepared, and 100 pieces of learning data were selected for each classification. The number of 241 classifications is arbitrary and can be selectively adjusted.

Images without text or other classifications were selected as much as possible, and if less than 100 images were filled, Google image crawling was used to fill in 100 images.

The 24,100 data sets created in this way are used for the image similarity measurement model, which is the first artificial intelligence model (first learning model), and after additional bounding box labeling, the second artificial intelligence model (second learning model), the classification estimation model was used for

Verification data production

The verification data consists of pairs of similar and dissimilar trademark images.

A pair of similar trademark images was created by parsing the Notice of Reason for Rejection by the Korean Intellectual Property Office. The reason for refusal in the statement of refusal is “A similar trademark image exists.” 292 similar pairs were produced by parsing the case of .

Dissimilar brand image pairs were randomly selected from 100,000 databases and produced. To match the number of similar pairs, 292 dissimilar pairs were produced.

first learning model

The first learning model includes a model for measuring the similarity of the overall shape of the figure. The first learning model is a model comprising a synthetic neural network, eg, inception, VGG. Any one of the models such as LeNet, AlexNet, ZFNet, VGG, GoogLeNet, ResNet, MobileNet, and EfficientNet can be used or some modified ones can be used. For similarity learning between images, a Siamese network, a contrastive loss, and a triplet loss can be used. In one embodiment, transfer learning using triplet loss is used in an existing image similarity measurement model. The triple loss training data was prepared as follows.

For example, by selecting two figure images from one classification of 241 classification, one as an anchor image, the other as a positive image, and by selecting one figure image from another classification as a negative image ( negative image). In each of the 241 classifications, the training data were selected in the same way.

By inputting the second trademark data stored in the database to the first learning model, it is possible to obtain the feature vector of the fully connected layer, which is the last layer. The feature vector of each trademark of the second trademark data is extracted and stored in the database. When a desired trademark (query trademark) is input to the first learning model, a feature vector of the query trademark is extracted, and the cosine similarity is calculated with the feature vector pre-stored in the database.

second learning model

The second learning model may include a model for classifying trademarks. The second learning model is a model that compares not only the similarity of the overall appearance of the figure, but also the similarity of the main part of the figure. In order to learn the second learning model, parts determined to be the main part of the figure are extracted from the figure included in the trademark. In one embodiment, the extraction of the feature may use a neural network and/or algorithm for object detection. For example, a model including a Region Proposal Network (RPN), a YOLO model, or the like may be used. In RPN, a plurality of object possibility boxes are held based on the point at which pixel values change rapidly, and the ratio of overlapping regions of interest (ROI) boxes of the boxes is calculated. If the mutual ROI is high, it is judged that the same features are shown, and 2 to 10 strong candidates are removed through this. The number of recessed parts can be arbitrarily selected by the user when training the learning model. One figure image includes a plurality of recesses, and each recess is classified by the second learning model.

9A to 9C are exemplary diagrams of classifying some images in a trademark image according to an embodiment.

For example, as shown in FIG. 9A , classification may be performed using some images as a main part in the entire image of a skeleton wearing glasses and a hat. In this case, classification of glasses, hats, and shades may be output from the image shown in FIG. 9A . Referring to FIG. 9B , a classification of a round crown, an insect, and a butterfly may be output from each of the partial images of the entire image. Referring to 9c, the classification of anchors and wings may be output.

In an embodiment, since the figure may include a plurality of main parts and each main part may belong to a different classification, one trademark may correspond to a plurality of classifications. In an embodiment, the second learning model may include a multi-label classification model. As training data of the second learning model, 24100 data of 241 classifications were used.

When learning to classify everything according to the figure classification table as the learning data, the classification may be biased. This may be because there are too many 241 classifications. Also, being biased is a kind of over-fitting problem. In an embodiment, a feature vector of one verification set (first set) has a value of a corresponding classification greater than a specific threshold value when not overfitting. In more detail, for example, when using N classifications, a feature vector has a shape of (N,1) and has a probability of each of the N classifications. Here, the probability that the feature vector has may be a predicted probability.

In one embodiment, when 241 classifications are used, a feature vector has the shape of (241,1) and contains the probability of each of the 241 classifications. A feature vector may be expressed as values of [a ₁ , ..., a ₂₄₁ ], and a ₁ , ..., a ₂₄₁ correspond to values of each classification. In the case of not overfitting, a value corresponding to the feature vector of the first set of classification appears large.

However, in the case of overfitting, a value of a corresponding classification may be smaller than a specific threshold and a value corresponding to another classification may be large. For example, although an image corresponding to the earth is input, a value of a feature vector corresponding to the sun may be greatly output.

This is because the sun and the earth have the same characteristic that they are 'round', so the two classifications may not be distinguished as they are learned. In this case, although the sun is classified as a sun, a problem arises that the earth is also classified as a sun. This case can be confirmed as a feature vector of the last layer.

More specifically, in the case of overfitting, most of the feature vectors of the 'verification set: earth' may have a low value corresponding to the earth classification. For example, you can take a low value around 0.3, and this value can be changed. In the case of overfitting, in the example of 'Verification Set: Earth', a value corresponding to another classification, for example, the Sun, may appear higher than a specific threshold.

Also, by comparing a feature vector of another verification set (the second set) with a feature vector of the first verification set, if the two feature vectors are similar, it is determined that the distinction between the two classifications is not clear. For example, if the feature vectors of the validation set:Sun are similar to those of the validation set:Earth, it is judged that the two classifications are not properly distinguished. The similarity of the feature vectors can be confirmed by the cosine similarity. This case is one example, and various cases may exist.

In the case of the above example, a sub-learning model is created by using only the trademark image classified as the earth and the trademark image classified as the sun as training data. A sub-learning model can be created using a multi-label classification model in a similar way to the main model.

Overfitting may occur between the two classifications (Earth, Sun) as above, but may occur in a larger number of classifications. In an embodiment, values of feature vectors of a plurality of classifications (M classifications, M being less than N) may not have accurate values due to overfitting.

For example, it may be determined that 'moon, earth, sun, circle, wheel, logo_1' has similarity. In this case, the sub-learning model is trained with 600 pieces of training data corresponding to 6 classifications, so that the 6 classifications are accurately learned. In this case, the feature vector is generated in the shape of (6,1) rather than (241,1).

According to an embodiment, the figure classification may use one main model and a plurality of sub-models. For example, as a result of learning by inputting second trademark data classified into N classifications into the main model, if any M classifications among the N classifications have similarities and are not accurately classified, trademark data corresponding to the M classifications is used to train the submodel. Since there may be a plurality of groups that are not accurately classified due to similarity, there may be a plurality of sub-models. In another embodiment, as a result of the sub-model learning using the trademark data corresponding to the M classifications, if any K classifications among the M classifications have similarities and are not accurately classified, another sub-model (of the sub-models) sub-model), another sub-model can be trained using trademark data corresponding to K classifications.

In an embodiment, the main model is a model that classifies all 241 classifications, and the sub-model is a model that classifies only specific classifications. The main model and sub-model are made using a multi-label classification model, but are models trained using other training data. As described above, in order to solve overfitting that occurs while performing 241 classification using the main model, a sub-model is trained using trademark data corresponding to a classification whose classification is not clear among 241 classifications. Accordingly, the sub-model may include a plurality of sub-models. Each sub-model is a model for classifying classifications that the main model does not accurately classify, and the above-mentioned two classifications (Earth, Sun) or 'Moon, Earth, Sun, Circle, Wheel, Logo_1' are examples.

The method of classifying trademark data using the main model and sub-model is as follows. Assume that the main model predicts [sun, person, mountain] for the input brand. It is determined whether any one classification of [sun, person, and mountain] is included in the classification type of the submodel. For example, if {moon, earth, sun, circle, wheel, logo_1} are the classifications classified by the first sub-model, {moon, earth, Since 'sun' is included among the sun, circle, wheel, and logo_1}, whether the classification of 'sun' is correct is performed by entering the input trademark into the first sub-model to perform detailed classification. The classification result of the first sub-model may replace the classification result of the main model.

For example, if the first sub-model classification result [Earth] is obtained, the final classification result becomes [Earth, person, mountain] instead of [Sun, person, mountain], and the classification result of the first sub-model [Earth] . [Sun, Man, Mountain] is maintained.

The classification predicted by the main model and the sub-model can be set to the top Nth. In an embodiment, the main model may predict the top three classifications, and each sub-model may be adjusted to predict the top two classifications. In this case, when [a, b, c] predicted by the main model are all included in the sub-model, it can be estimated as a maximum of 6 classifications [a1, a2, b1, b2, c1, c2]. The number of upper classifications predicted by the main model and the sub-model is exemplary and can be changed.

Accordingly, in an embodiment, 0 to 6 classifications may be estimated per one image. In this case, 0 classification means no classification. As with the first learning model, the classifications of the second learning data are stored in the database.

The classification can later be used to determine the degree of similarity of the final trademark.

Trademark Search

The final similarity is calculated using the results of the image similarity measurement model (first learning model) and the classification estimation model (second learning model). It is a method of multiplying the overlapping ratio of the classification of the input image predicted by the classification estimation model and the classification of the images in the database by the image similarity.

When the user inputs an image, the final model calculates the similarity with the images in the database, and displays them in order of the highest similarity.

Trademark search experience

1 is a block diagram of a trademark search environment according to one embodiment.

Referring to FIG. 1 , the trademark search environment may include a terminal 100 , a trademark search server 200 , and a database 300 . The terminal 100 is a terminal 100 of a user who wants to search for a trademark. The terminal 100 may perform a trademark search by accessing the trademark search server 200 through wired or wireless communication. The terminal 100 may include a smart phone, a laptop computer, a tablet PC, an ultrabook, a wearable device, and the like. The user may set a search target through editing such as reducing or enlarging a trademark or extracting, reducing, or enlarging only a part of the trademark through the terminal 100 .

The trademark search server 200 may include a separate cloud server or computing device. In addition, the trademark search server 200 may be a neural network system installed in the processor of the terminal 100 or the data processing unit of the database 300 . That is, although the database 300 and the trademark search server 200 are illustrated as separate devices in FIG. 1, they may be a single device.

The database 300 may be configured to store at least one of learning data and a feature vector of a trademark. The trademark feature vector may be obtained by inputting trademark data after training the trademark search server 200 . The database 300 may include a communication module and a data processing unit for communicating with the terminal 100 and the trademark search server 200 .

2 is a block diagram of a terminal 100 according to an embodiment.

Referring to FIG. 2 , the terminal 100 may include an input unit 110 , a communication unit 120 , a memory 130 , a display 140 , and a processor 150 .

Through the input unit 110 , the user may perform setting and execution input for various functions of the terminal 100 . The input unit 110 includes a touch input unit (eg, a touch sensor, a touch key, a physical key, etc.) sensing a user's touch and a microphone sensing a voice input. A user input may be sensed by including at least one of them.

The terminal 100 may communicate with the trademark search server 200 and the database 300 through the communication unit 120 . For example, the communication method of the communication unit 120 is GSM (Global System for Mobile communication), CDMA (Code Division Multi Access), HSDPA (High Speed Downlink Packet Access), HSUPA (High Speed Uplink Packet Access), LTE (Long) Term Evolution), LTE-A (Long Term Evolution-Advanced), etc.), WLAN (Wireless LAN), Wi-Fi (Wireless-Fidelity), Wi-Fi (Wireless Fidelity) Direct, DLNA (Digital Living Network Alliance), WiBro A network constructed according to (Wireless Broadband) and WiMAX (World Interoperability for Microwave Access) may be used, but the present invention is not limited thereto, and may include all transmission method standards to be developed in the future.

The memory 130 may store a plurality of application programs or applications driven in the terminal 100 , data for operation of the terminal 100 , and commands. For example, in terms of hardware, the memory 130 may be various storage devices such as ROM, RAM, EPROM, flash drive, hard drive, etc., and the memory 130 stores the memory 130 on the Internet. It may be a web storage that performs a function.

The display 140 includes a touch screen, a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. It may include at least one of a flexible display and a three-dimensional display (3D display).

The processor 150 is configured to control each component of the terminal 100 . For example, the processor 150 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and controllers. ), micro-controllers, microprocessors, or any other type of processor for performing other functions.

3 is a block diagram of a trademark search server 200 according to an embodiment.

Referring to FIG. 3 , the trademark search server 200 stores a communication unit 310 communicating with the terminal 100 and the database 300 , a data processing processor 330 , and an artificial neural network for extracting a feature vector from a trademark. It may include a storage unit 320 to The processor 330 may retrieve and execute the artificial neural network. The artificial neural network may include a first learning model and a second learning model. In an embodiment, the processor 330 may be configured to include a main processor that controls each configuration and a graphics processing unit (GPU) that performs an operation necessary for driving the neural network.

Trademark Search Environment Validation

The verification data was produced, the verification data was input to each of the first learning model and the second learning, and the degree of similarity was measured.

The validation data includes pairs of similar and dissimilar trademark images. A pair of similar trademark images was created using the Notice of Grounds for Rejection. In one embodiment, the reason for rejection of the notification of reasons for rejection is parsed, and when "a similar trademark image exists" among the reasons for rejection exists, it is classified as a pair of similar trademark images. This made 292 similar image pairs. 292 pairs were randomly extracted from the second training data with dissimilar trademark image pairs.

The accuracy of dividing similarity/dissimilarity based on a certain threshold similarity was obtained by inputting verification data into the first learning model.

4A is a similarity histogram of a similarity image using the first learning model according to an embodiment, and FIG. 4B is a similarity histogram of a random image using the first learning model according to an embodiment. Accuracy was 80.9%, recall was 85.9%, precision was 78.2%, and F1 score was 80.8%.

5A is a similarity histogram of a similarity image using a first learning model and a second learning model according to an embodiment, and FIG. 5B is a similarity histogram of a random image using the first learning model and a second learning model according to an embodiment.

Verification data was input to each of the first learning model and the second learning model. This is a method of multiplying a weight according to the overlapping ratio of the classification of the input image output by the second learning model and the classification of the images in the database with the image similarity output of the first learning model. When the user inputs an image, the final model calculates the similarity with the images in the database, and displays them in order of the highest similarity. The accuracy was 83.2%, the recall was 87.6%, the precision was 80.2%, and the F1 score was 83.8%. About 3% improvement compared to using only the first learning model.

Trademark search model according to an embodiment

6 is a block diagram illustrating a trademark search model 600 according to an embodiment. The disclosure of the above specification applies to the trademark search model 600 . Although not shown in FIG. 6 , there may be a separate processor for executing the trademark search model 600 and/or a storage unit for storing the trademark search model 600 .

When a search target trademark is input to the trademark search model 600 , the trademark search model 600 searches for a similar trademark image. When inputting a search target trademark, the user may reduce or enlarge at least a portion of the trademark. Alternatively, the user may select only at least a part of the trademark as a search target.

The trademark search model 600 includes a first learning model 610 and a second learning model 620 . The first learning model 610 is configured and trained to measure the similarity of figures included in a trademark, and may include a neural network model using artificial intelligence. The first learning model 610 may be configured to be trained with a triple loss function. When a trademark is input to the first learning model 610, the first learning model 610 provides a feature vector (hereinafter, referred to as a first feature vector) of a figure included in the input trademark (query trademark image), and a trademark pre-stored in the database. At least one of a feature vector of (hereinafter, referred to as a second feature vector) and a cosine similarity of the first feature vector may be output.

The second learning model 620 is configured and trained to classify figures included in the trademark into a plurality of classifications, for example, N classifications, and may include a neural network model using artificial intelligence. The second learning model 620 may be trained using the same training data as the first learning model or may be trained using different training data. In an embodiment, the second learning model may include at least one of an object detection model, an algorithm, and a neural network. The second learning model 620 may classify one brand into at least one classification.

The second learning model 620 may include a main model 622 and at least one sub-model 624 and 626 . The main model 622 is trained using N classified training data, and the first and second sub-models 624 and 626 are to be trained using trademark data corresponding to M classifications selected from among the N classifications. can The first and second sub-models 624 and 626 may be trained using trademark data corresponding to different classifications. The first and second sub-models 624 and 626 may be models for re-classifying trademark data that the main model 622 has not clearly classified. In an embodiment, the classification classified by the first and second sub-models 624 and 626 may be predetermined. For example, the main model 622 classifies the query image into the first to 100th classifications, the first sub-model 624 classifies the query image into the first to fourth classifications, and the second sub-model 626 The query image may be classified into the 70th to the 75th classification. The above classification is merely exemplary.

For example, when a query trademark image is input to the main model 622 , the main model 622 classifies the query trademark image into 100 classifications and outputs a top specific number of classifications. That is, a feature vector in the form of (100, 1) and a predicted probability value of [a ₁ , ..., a ₁₀₀ ] may be output, and the top three classifications of the predicted probability values may be output. The main model 622 may select and output only a classification having a specific value or more. For example, only a case in which the predicted probability value is 0.3 or more may be output, and accordingly, all of the top three may not be output.

When a query trademark image is input to the first and second sub-models 624 and 626 , the first and second sub-models 624 and 626 output a higher specific number of classifications. That is, the top two classifications can be output. In an embodiment, the first and second sub-models 624 and 626 may select and output only a classification having a specific value or more. For example, the first sub-model 624 uses the first to fourth classifications as feature vectors of the form (4, 1) and predicted probability values (classification values) of [a ₁ , ..., a ₄ ], The second sub-model 626 may output the 70th to 75th classifications as feature vectors of the form (6, 1) and predicted probability values (classification values) of [a ₇₀ , ..., a ₇₅ ]. The first and second sub-models 624 and 626 can select and output only the classification having a specific value or more. For example, only when the predicted probability value is 0.3 or more, it can be output, and accordingly, the top two are all output. it may not be

In one embodiment, when any one of the classifications output by the main model 622 corresponds to the classification classified by the first and second sub-models 624 and 626, the first and second sub-models 624 and 626 ), the query trademark image is input, and the first and second sub-models 624 and 626 classify the query trademark image once more.

For example, the classification output by the main model 622 may be the second classification, the twentieth classification, and the 55th classification ([A2, A20, A55]). Since the second classification is a classification classified by the first sub-model 624 , the query trademark image is input to the first sub-model 624 and classified as at least one of the first to fourth classifications. For example, the query trademark image may be classified into a first classification and a fourth classification ([A1, A4]) by the first sub-model 624 . Accordingly, the second classification is replaced by the first classification and the fourth classification. (However, a classification greater than or equal to a specific value among the first classification and the fourth classification may be selected.) Accordingly, the second learning model 620 applies the query trademark image to the first classification, the fourth classification, the 20th classification, and the 55th classification. It can be classified by classification ([A1, A4, A20, A55]).

The trademark search model 600 determines the final trademark similarity based on the weight according to the classification output by the second learning model 620 and the cosine similarity between the second feature vector and the first feature vector output by the first learning model 610 . and calculates the final similarity of the trademark image accordingly. In an embodiment, the weight may be determined in consideration of the number of overlapping the classification of the input image predicted by the second learning model 620 and the classification of the images in the database.

For example, if the number of overlapping the classification of the input image predicted by the second learning model 620 and the classification of images in the database is 3 or more, 1 point, 2 times, 0.95 points, 1 time, 0.9 points, 0 times A weight can be set, such as 0.8 points. These weights may be changed based on the final similar trademark selection result. These weights are exemplary and can be changed.

In addition, the weight may be adjusted by performing correction in consideration of the number of output classifications (predicted classifications). For example, when the number of output classifications is small, it can be determined that the classification is correct, and a correction constant can be added to the weight.

For example, the classification output by the second learning model 620 for the first query image is the first classification, the second classification, and the third classification ([A1, A2, A3]), and the classification of the trademark image stored in the database is the first classification, the second classification, and the third classification ([A1, A2, A3]), the number of overlapping is three and the weight is one.

The classification output by the second learning model 620 for the second query image is the first classification, the second classification ([A1, A2]), and the classification of the trademark image stored in the database is the first classification, the second classification ([A1, A2]) A1, A2]), since the number of overlaps is 2, the weight is 0.95, but since all classifications overlap, it can be determined that the weight should be 1 as in the case above. Therefore, in this case, the weight can be adjusted to 1 by correcting the weight.

That is, the weight of the overlapping number may vary according to the number of classifications of the query image output by the second learning model 620 .

Also, even if the number of overlapping classifications is the same, there may be cases in which the weights are adjusted. For example, the classification output by the second learning model 620 for the query image is the first classification and the second classification ([A1, A2]), and the classification of the trademark image stored in the database is the first classification and the second classification , for the fourth classification ([A1, A2, A4]), the number of overlaps is 2 and the weight is 0.95. On the other hand, if the classification of other trademark images stored in the database is the first classification and the second classification ([A1, A2]), the number of overlaps is the same as above, but since all classifications overlap, the similarity is higher, and therefore the weight is 0.95 can be considered larger. Accordingly, by correcting the weight (adding a correction constant) at this time, the weight can be adjusted to 1.

In this way, the weight may be corrected in consideration of the number of classifications output by the second learning model 620 and/or the number of classifications of trademark images stored in a database to be compared.

The determined weight is multiplied by the cosine similarity of the first learning model 610, and a final similarity value can be obtained using this. The second learning model may be a model that complements the first learning model. Weights and correction constants can be appropriately adjusted to exclude cases where there are many features, but each feature does not significantly affect the similarity of the trademark.

In one embodiment, the trademark search model 600 may calculate the cosine similarity and the final similarity, but the first learning model 610 outputs the first feature vector and the second learning model 620 performs only classification and , cosine similarity and final similarity may be calculated in a separate processor.

A trademark search method according to an embodiment

7 is a flowchart illustrating a trademark search method 700 according to an embodiment. The trademark search method 700 may be performed by the aforementioned trademark search model 600 .

When a search target trademark (query image) is input into the trademark search model 600, a search for a trademark having a similar image is started. In an embodiment of the present invention, the query image is input to the first learning model 610 and the second learning model 620, respectively (S705 and S710). When inputting a query image, the user may input the entire image, and may reduce, enlarge, or select at least a portion of the query image to input.

The cosine similarity between the feature vector of the query image and the feature vector of the trademark stored in the database is calculated (S720). In more detail, the feature vector of the query image is extracted, the feature vector of the trademark previously stored in the database is called, and the cosine similarity between the feature vector of the query image and the feature vector of the trademark stored in the database is calculated.

The query image input to the second learning model 620 is first input to the main model (S715). The main model 622 classifies the input query image, and determines whether the classification output by the main model 622 is included in the classification classified by at least one of the sub-models 624 and 626 ( S725 ).

When at least one of the classifications output by the main model 622 is included in the classification classified by the sub-models 624 and 626, a query image is input to the corresponding sub-models 624 and 626 (S730). Classification of the query image is predicted based on the output result of the main model 622 and the output result of the sub-models 624 and 626 ( S735 ). In an embodiment, when the classification output by the main model 622 and the classification classified by the sub-models 624 and 626 overlap, the overlapped classification may be replaced with the classification output by the sub-models 624 and 626. have.

The classification of the query image predicted by the second learning model 620 is compared with the classification of the images in the database (S740). The number of overlapping of the classification predicted by the second learning model 620 and the classification of each image stored in the database may be obtained.

A weight is calculated based on the number of overlapping the classification predicted by the second learning model 620 and the classification of each image stored in the database (S745). In an embodiment, the weight may be corrected based on the number of classifications predicted by the second learning model 620 and/or the number of classifications of images stored in the database.

A final similarity is calculated based on the cosine similarity and the weight calculated in S745 (S750).

Similar trademarks are displayed based on the final similarity (S755).

Database production according to an embodiment

8 is a flowchart illustrating a method 800 for creating a database according to an exemplary embodiment. The database production method 800 may include a method of training the above-described trademark search model 600 .

Training data for training the trademark search model is produced (S805). In one embodiment, the learning data is provided from Cypris Plus (plus.kipris.or.kr), to remove duplicate trademarks, to remove trademarks consisting of only characters, and to distinguish characters from composite trademarks to image (figure) trademarks Only trademarks classified as The extracted trademark data is divided into N classifications, and a specific number of trademarks are randomly extracted for each classification to produce learning data classified into N classifications.

The produced learning data is input to the trademark search model 600 . That is, learning data is input to the first learning model 610 and the second learning model 620 ( S810 and S815 ). The first learning model 610 is trained through a triple loss function. An anchor image and a positive image are selected from a first classification among N classifications, and a negative image is selected from a second classification different from the first classification among the N classifications. 1 Train the learning model.

When the learning of the first learning model is completed, trademark data classified as an image (figure) trademark is input to the first learning model (S820). Accordingly, it is possible to extract the feature vector of each brand.

The second learning model 620 may include a main model 622 and at least one sub-model 624 and 626 . The training data is input to the main model 622 (S825). The main model performs learning by classifying the training data into N classifications. At least one processor checks the learning result (S830).

It is checked whether overfitting is based on the learning result (S835). In one embodiment, it is checked that N classifications are performed correctly. As the size of the N number increases, the accuracy of classification may decrease. As an example of overfitting, there may be a case in which classification accuracy of specific classifications among N classifications is low.

For example, the feature vectors of N classifications may have a shape of (N, 1) and have values of [a ₁ , ... a _N ]. When classification is performed correctly, a value corresponding to the classification (hereinafter referred to as a classification value) appears large. For example, in the trademark image belonging to the first classification, the classification value of a ₁ is the largest among [a ₁ , ... a _N ]. However, the difference between the classification values may not be greater than a specific value, or a value belonging to a different classification may be larger than a specific value or similar enough to be difficult to be classified.

For example, in the trademark image belonging to the first classification, the value of a ₁ among [a ₁ , ... a _N ] should appear the largest, but other classification values may appear large. For example, when a value corresponding to another classification, such as a ₆ , a ₁₀ , a ₂₀ , etc. instead of a ₁ , as a learning result of the training data belonging to the first classification is greater than a predetermined specific value, or a ₆ , a ₁₀ , a ₂₀ When the value is smaller than a ₁ but greater than a specific value, it may be determined that the values of a ₁ , a ₆ , a ₁₀ , and a ₂₀ are similar. In this case, since the first classification cannot be accurately classified with the values of a ₁ , a ₆ , a ₁₀ , and a ₂₀ , the training data corresponding to the first classification, the sixth classification, the tenth classification, and the 20th classification are applied to the sub-model. input (S840). Accordingly, a sub-model for accurately classifying the first classification, the sixth classification, the tenth classification, and the twentieth classification is trained (S840).

The first classification, the sixth classification, the tenth classification, and the 20th classification cannot be distinguished, and as a result of examining the learning results of the sixth classification, the classification value of the sixth classification may not be distinguished from the classification values of other classifications. For example, if a ₃₀ is a large value instead of a ₆ as a result of the learning of the 6th classification or is greater than a specific value, the 30th classification is also applied to the training data corresponding to the 1st, 6th, 10th, and 20th classifications. In addition, it can be training data for training the sub-model. Alternatively, an additional sub-model for classifying the sixth classification and the thirtieth classification may be trained.

At least one sub-model is trained according to overfitting, and finally, trademark data is input to the second learning model ( S845 ). The second learning model outputs a classification corresponding to each brand from the trademark data.

The database receives the feature vectors of each brand extracted by the learning of the first learning model and the classification output by the second learning model, and the database is produced (S850).

The apparatus and method described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

Software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The described embodiments of the present disclosure may also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to a person skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

A computing device may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s), via wired and/or wireless communications. The remote computer(s) may be workstations, server computers, routers, personal computers, portable computers, microprocessor-based entertainment devices, peer devices, or other common network nodes, typically comprising the components described for the computing device. includes many or all of The logical connections shown include wired/wireless connections to a local area network (LAN) 652 and/or a larger network, eg, a wide area network (WAN) 654 . Such LAN and WAN networking environments are common in offices and companies, and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a worldwide computer network, for example, the Internet.

As described above, although the embodiments have been described with reference to the limited drawings, those skilled in the art may apply various technical modifications and variations based on the above. For example, the described techniques are performed in an order different from the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

It is possible to provide a trademark search model and a learning method that can accurately search for a trademark similar to a trademark to be searched.

100: terminal 110: input unit
120: communication unit 130: memory
140: display 150: processor
200: trademark search server 300: database
310: communication unit 320: storage unit
330: processor 600: trademark search model
610: first learning model 620: second learning model
622: main model 624: first sub-model
626: second sub model

Claims (12)

receiving, by a first learning model executed by one or more processors, a query image;
extracting, by the first learning model, a first feature vector related to the overall appearance of the query image;
calculating, by the one or more processors, a degree of similarity between a second feature vector of a trademark image stored in a database and the first feature vector;
receiving, by a second learning model executed by the one or more processors, the query image, wherein the second learning model includes a main model and a first sub-model, wherein the main model divides the query image into N classifications. is configured to classify, the first sub-model is configured to classify the query image into M classifications selected from among the N classifications, and the classification by the main model and the first sub-model is a classification of the same layer;
outputting, by the second learning model, at least one classification corresponding to the query image based on the partial image of the query image;
selecting at least one image similar to the query image from among trademark images stored in the database based on the similarity between the second feature vector and the first feature vector and the classification of the second learning model; and
providing the selected image;
The outputting of the at least one classification by the second learning model may include: providing, by the main model, a first classification based on the query image; and providing, by the first sub-model, a second classification based on the query image. According to claim 1,
The step of outputting the at least one classification by the second learning model comprises:
and replacing the first classification with the second classification. The step of outputting the at least one classification by the second learning model comprises:
and in response to determining, by the one or more processors, that the first classification and the second classification are different, replacing the first classification with the second classification. According to claim 1,
The second learning model further comprises a second sub-model,
the second sub-model is configured to classify the query image into K classifications selected from among the N classifications, wherein the selected M classifications and the selected K classifications are different, and the classification classified by the second sub-model is the It is a classification of the same layer as the classification classified by the main model and the first sub-model.
The step of outputting the at least one classification by the second learning model further comprises providing, by the second sub-model, a third classification based on the query image. According to claim 1,
The step of outputting the at least one classification by the second learning model comprises:
before the first sub-model providing the second classification based on the query image, determining, by the one or more processors, whether the first classification is included in the M classifications; and in response to determining that the first classification is included in the M classifications, the first sub-model providing the second classification based on the query image. According to claim 1,
selecting at least one image similar to the query image from among trademark images stored in the database based on the similarity between the second feature vector and the first feature vector and the at least one classification,
determining the number of overlapping classifications by comparing the at least one classification with the classifications of trademark images stored in the database;
retrieving weights according to the number of overlapping classifications; and
and selecting at least one image similar to the query image based on a product of a similarity between the second feature vector and the first feature vector and the weight. According to claim 1,
selecting at least one image similar to the query image from among trademark images stored in the database based on the similarity between the second feature vector and the first feature vector and the at least one classification,
determining the number of overlapping classifications by comparing the at least one classification with the classifications of trademark images stored in the database;
retrieving a weight based on the number of classifications overlapping with at least one of the number of the at least one classification and the number of classifications of the trademark image stored in the database; and
and selecting at least one image similar to the query image based on the weight and the similarity between the second feature vector and the first feature vector. storage configured to store the trademark search model; and
a trademark retrieval device comprising a processor configured to execute the trademark retrieval model;
The trademark retrieval model includes a first learning model and a second learning model, wherein the first learning model extracts a first feature vector from a query image so that the processor determines the similarity with a second feature vector of the trademark image stored in the database. configured to obtain, the second learning model is configured to output a plurality of classifications corresponding to the partial images based on the partial images of the query image;
In the trademark search apparatus, the processor is configured to select at least one image similar to the query image from among the trademark images stored in the database based on a degree of similarity between the second feature vector and the first feature vector and the plurality of classifications,
The second learning model includes a main model, a first sub-model and a second sub-model, the main model is configured to classify the query image into N classifications, and the first sub-model divides the query image into the N classifications. is configured to classify into M classifications selected from among N classifications, and the second sub-model is configured to classify the query image into K classifications selected from among the N classifications - The selected M classifications and the selected K classifications are different - and the classification classified by the main model, the first sub-model, and the second sub-model is a classification of the same layer. 9. The method of claim 8,
The main model provides a first classification based on the query image,
the one or more processors are configured to determine whether the first classification is included in any one of the selected M or the selected K classifications;
wherein the second sub-model or the second sub-model is configured to provide a second classification based on the query image in response to determining that the first classification is included in the selected M or K selected classifications. Device. 9. The method of claim 8,
The processor is
By comparing the plurality of classifications with the classifications of trademark images stored in the database, the number of overlapping classifications is determined, and weights according to the number of overlapping classifications are called by comparing the plurality of classifications with the classifications of the trademark images stored in the database, , configured to select at least one image similar to the query image based on a product of a similarity between the second feature vector and the first feature vector and the weight. 9. The method of claim 8,
The processor is
The number of overlapping classifications is determined by comparing the plurality of classifications with the classifications of the trademark images stored in the database, and at least one of the number of the plurality of classifications and the number of classifications of the trademark images stored in the database and the overlapping classifications and fetch a weight based on the number, and select at least one image similar to the query image based on the weight and a degree of similarity between the second feature vector and the first feature vector. 9. The method of claim 8,
wherein the processor replaces the classification classified by the main model with the classification classified by the first sub-model or the second sub-model so that a second learning model outputs the plurality of classifications.

Images