KR20230032536A - Methods for automatically identifying a match between a product image and a reference drawing based on artificial intelligence - Google Patents

Abstract

A method for evaluating similarity between a product image and a reference drawing comprises the steps of: acquiring one or more feature vectors corresponding to one or more images of a product; searching for a plurality of training models constructed to determine the similarity of each reference product; and applying each of the training models to a feature vector among the one or more feature vectors corresponding to the one or more images of the product to determine similarity between one image among the one or more images of the product and the image of each reference product for each training model among the plurality of training models, which have been constructed to determine similarity between the product and each reference product; and providing the similarity between the product and each reference product. Provided is also a method for constructing the model.

Description

Matching automatic identification method between product image and reference image based on artificial intelligence

The present disclosure relates generally to methods for registering product images and reference images, and more particularly to systems and methods for registering product images and patent drawings.

Determining the registration between two images is an important task in image processing technology. In particular, electronic devices capable of identifying similar images have many potential applications.

Although there have been advances in image registration techniques, these techniques do not work well with specific drawings, such as engineering drawings (eg computer aided design drawings) or patent drawings.

Accordingly, a need exists for a tool that can accurately determine the registration between a product image and these reference drawings. The methods and devices described herein address this need. These methods and devices may replace conventional methods and devices. Alternatively, these methods and devices may complement conventional methods and devices.

According to some embodiments, a method of evaluating similarity between a product image and a reference drawing is performed in a computer device having one or more processors and a memory storing one or more programs configured for execution by the one or more processors. The method includes obtaining one or more feature vectors corresponding to one or more images of a product; retrieving a plurality of training models built to determine each similarity of each reference product; Determining a similarity between one of the one or more images of a product and an image of each reference product, for each training model, of the plurality of training models built to determine respective similarities between the product and each reference product applying each training model to feature vectors of one or more feature vectors corresponding to one or more images of the product; and providing respective similarities between the product and each reference product.

According to some embodiments, a computer readable storage medium stores one or more programs for execution by a computer system having one or more processors and memory. The one or more programs obtain one or more feature vectors corresponding to one or more images of the product; retrieve a plurality of training models built to determine each similarity of each reference product; Determining a similarity between one of the one or more images of a product and an image of each reference product, for each training model, of the plurality of training models built to determine respective similarities between the product and each reference product apply each training model to feature vectors of one or more feature vectors corresponding to one or more images of the product; and instructions for providing respective similarities between the product and each reference product.

According to some embodiments, a computer system includes one or more processors and memory. When the memory is executed by one or more processors, the one or more processors obtain one or more feature vectors corresponding to one or more images of the product; retrieve a plurality of training models built to determine each similarity of each reference product; Determining a similarity between one of the one or more images of a product and an image of each reference product, for each training model, of the plurality of training models built to determine respective similarities between the product and each reference product apply each training model to feature vectors of one or more feature vectors corresponding to one or more images of the product; and stores one or more programs containing instructions to provide respective similarities between the product and each reference product.

According to some embodiments, a method of building a model for evaluating similarity between a product image and a reference drawing is performed in a computer device having one or more processors and a memory storing one or more programs configured for execution by the one or more processors. The method includes, for each reference product of a plurality of reference products, retrieving images of each reference product; and constructing each feature vector for each image of the retrieved images of each reference product. The method also includes training a model using at least a subset of feature vectors to determine similarity of each reference product; and storing the trained model in a database for subsequent use in determining the similarity of each reference product.

According to some embodiments, a computer readable storage medium stores one or more programs for execution by a computer system having one or more processors and memory. The one or more programs retrieve, for each reference product of the plurality of reference products, images of each reference product; and instructions for constructing each feature vector for each image of the retrieved images of each reference product. The one or more programs also train a model using at least a subset of the feature vectors to determine similarity of each reference product; and instructions for storing the training model in a database for subsequent use in determining the similarity of each reference product.

According to some embodiments, a computer system includes one or more processors and memory. When the memory is executed by one or more processors, the one or more processors, for each reference product of the plurality of reference products, retrieves an image of each reference product; and stores one or more programs including instructions for constructing each feature vector for each image of the retrieved images of each reference product. The one or more programs, when executed by the one or more processors, also cause the one or more processors to train a model using at least a subset of the feature vectors to determine similarity of each reference product; and instructions for storing the training model in a database for subsequent use in determining the similarity of each reference product.

Accordingly, methods and systems for building (eg, training) models for similarity determination between a product image and a reference figure and using the training models for similarity determination between a product image and a reference figure are disclosed. Such methods and systems facilitate identification of one or more products that may appear similar to referenced drawings, including automatically identifying products that may infringe third party intellectual property rights (eg, design patents, trade dress, etc.) can be used to do

Both the foregoing general description and the following detailed description are illustrative and explanatory, and are intended to provide further explanation of the claimed disclosure.

For a better understanding of such systems, methods and graphical user interfaces, as well as additional systems, methods and graphical user interfaces shown in association with the drawings, reference is made to the following description of implementations in conjunction with the drawings, where like reference numbers refer to the drawings. It means the corresponding part of the whole.
1A illustrates training one or more models in accordance with some implementations.
1B illustrates the use of one or more models according to some implementations.
2A is a schematic diagram illustrating a computer device in accordance with some implementations.
2B is a block diagram illustrating a server according to some implementations.
3A-3B illustrate a model training method according to some implementations.
4A-4B provide a flow diagram of a method for similarity assessment between a product image and reference drawings, in accordance with some implementations.
5A-5B provide a flow diagram of a method of building a model for similarity assessment between a product image and reference drawings, according to some implementations.
6 is a schematic diagram illustrating two operations involved in matching identification in accordance with some implementations.
Reference is made to implementations, examples of which are illustrated in the accompanying drawings. In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present disclosure may be practiced without requiring these specific details.

1A illustrates training one or more model(s) 132 using drawing data 120 from a plurality of drawings 110 (eg, reference drawings such as engineering drawings or patent drawings). The drawing data 120 is a machine learning engine configured to train (eg, produce, generate) one or more models 132 to determine similarity with one or more of the plurality of drawings 110. (130) is entered.

In some embodiments, multiple drawings include drawings 110 of the same type. For example, the plurality of drawings may include drawing 110 which is all a black and white drawing. In another example, the plurality of drawings may include drawing 110, which is all a color drawing. The type of drawing in the plurality of figures determines the type on which one or more model(s) 132 are trained to provide similarity value(s). For example, when the plurality of drawings include black-and-white drawings, one or more model(s) trained using drawing data 120 corresponding to the plurality of drawings 110 (eg, the plurality of black-and-white drawings). ) 132 is trained to give the similarity value(s) of a particular black and white drawing. In some implementations, the type of drawing also identifies whether the drawing is a schematic or photographic and/or whether the drawing includes shading.

Drawing data 120 for each drawing in a plurality of drawings is obtained. Drawing data 120 includes image data 122 and image type 124 . In some implementations, drawing data 120 also includes other data 126 . For example, the first drawing data 120-1 (eg, medical data or medical information) is obtained for the first drawing 110-1. Drawing data 120-1 includes image data 122-1, image type 124-1 and, optionally, other data 126-1 corresponding to the first drawing 110-1.

Image data 122 includes information representing graphic elements of an associated figure. For example, the image data 122 may include information indicating color (or black and white) of a plurality of pixels. In some implementations, image data 122 is compressed. In some implementations, image data 122 is uncompressed. Image types 124 include types of drawings. For example, the image type may include information such as black-and-white or color, a contour diagram or a filled drawing, a drawing with shading, or a drawing without shading. Other data 126 includes information identifying the associated product or associated patent information (eg, the license number of the patent from which the drawing was obtained).

Machine learning engine 130 uses image data 122 and, optionally, image type 124 corresponding to each figure to construct a feature vector for each figure in the plurality of figures. Machine learning engine 130 then uses the feature vectors to train one or more models 132 so that model 132 can determine similarity to one or more figures.

In some implementations, the one or more models 132 include a plurality of models (eg, a first model and a second model) and each model of the one or more models 132 is trained to provide similarity values for product images. do. For example, a first model can be trained to provide similarity values for product images of a first image type and a second model can be trained to provide similarity values for product images of a second image type different from the first image type. can be trained For example, a first image type may include a black and white floor plan and a second image type may include a black and white perspective view.

In some implementations, the first model is trained using a plurality of first figures and the second model is trained using a plurality of second figures different from the first plurality of figures. For example, the first plurality of figures differs from the second plurality of figures by at least one figure. In some implementations, the first plurality of figures includes one or more figures in common with the second plurality of figures.

Additional details regarding training one or more models 132 are provided with respect to FIG. 3A.

1B illustrates using one or more training models 132 trained to determine similarity value(s). New drawing data 141 corresponding to the new drawing 140 is provided to the model(s) 132 . The new drawing data 141 includes an image type 144 and image data 142 corresponding to the new drawing 140 . In some implementations, the new drawing data 141 also includes other data 146 corresponding to the new drawing 140 . New drawing data 141 is provided as input to one or more training models 132, which output results 151 for one or more other reference drawings. One or more training models 132 output a plurality of similarity values 152 .

FIG. 2A is a block diagram illustrating a computer device 200 , which is a computer system capable of training and/or running model(s) 132 in accordance with some implementations. Various examples of computer device 200 include desktop computers, laptop computers, tablet computers, server computers, server systems, wearable devices such as smart watches, and desktop computers having processors capable of training and/or operating model(s) 132 . Other computer devices are included. Computer device 200 may be a data server hosting one or more databases, models or modules, or may provide various executable applications or modules. Computer device 200 generally includes one or more processing units (processors or cores) 202, one or more network or other communication interfaces 204, memory 206, and one or more communication buses (or 208). Communication bus 208 optionally includes circuitry (sometimes referred to as a chipset) that interconnects and controls communications between system components. Computer device 200 generally includes a user interface 210 . User interface 210 generally includes a display device 212 (eg, a screen or monitor). In some implementations, computer device 200 includes input devices such as a keyboard, mouse, and/or other input buttons 216 . Alternatively, or also, in some implementations, display device 212 includes a touch-sensitive surface 214, where display device 212 is a touch-sensitive display. In some implementations, the contact-sensitive surface 214 responds to various swipe gestures (eg, vertical and/or horizontal continuous gestures) and/or other gestures (eg, single/double taps). is set to detect For computing devices having a touch-sensitive surface 214, a physical keyboard is optional (eg, a soft keyboard may be displayed when keyboard input is required). The user interface 210 also includes an audio output device 218, such as a speaker or an audio output connection device connected to a speaker, earphone, or headphone. In addition, some computer devices 200 use voice recognition software and audio input devices 220, such as microphones, to supplement or replace keyboards. An audio input device 220 (eg, a microphone) captures audio (eg, a user's speech).

Memory 206 may be a high-speed random-access memory device such as DRAM, SRAM, DDR RAM, or random-access solid-state memory device using other semiconductors. ); It may include non-volatile storage devices such as one or more magnetic disk storage devices, optical disk storage devices, flash storage devices, or non-volatile solid-state storage devices using other semiconductors. In some implementations, memory 206 includes one or more storage devices located remotely from processor 202 . Memory 206 or, non-volatile storage device within memory 206 includes non-transitory computer-readable storage media. In some implementations, memory 206 or a computer readable storage medium of memory 206 stores the following programs, modules, and data structures, or a subset or superset thereof:

- operating system 222, which includes procedures for handling various basic system services and performing hardware dependent tasks;

- a communication module 224 used to connect the computer device 200 to other computers and devices via one or more communication network interfaces 204 (wired or wireless), such as the Internet, other wide area networks, local area networks, metro networks;

- a web browser 226 (or other application capable of displaying web pages) that allows users to communicate with remote computers or devices over a network;

- an audio input module 228 (e.g. a microphone module) that processes the audio captured by the audio input device 220; The captured audio may be transmitted to a remote server and/or processed by an application running on computer device 200 (eg, modeling application 230);

- access and edit drawing data 120 including image data 122, image type 124 and other data 126 and/or image data 142, image type 144 and other data 146 A modeling application 230 including a graphical user interface 100 that allows a user to navigate the modeling application 230, such as accessing and editing new drawing data 141 including ). For example, one or more users may use graphical user interface 100 of modeling application 230 to select a subset of reference figures. Drawing data 120 is then compiled via machine learning engine 130 to train model(s) 132 . The modeling application 230 may input new drawing data 141 into the model(s) 132 and utilize the model(s) 132 to determine a similarity value. Model(s) 132 takes drawing data (including image data 142, image type 144, and other data 146) into account when generating similarity values;

- a data processing module 232 configured to perform the necessary preprocessing steps to convert the raw information into the correct data type for drawing data 120 or new drawing data 141; For example, the data processing module 232 may be set to perform image processing for converting a color drawing into a black and white drawing or converting a view of the drawing. Data processing module 232 may be configured to perform one or more calculations based on the received raw data to generate data values for drawing data 120 or new drawing data 141 . The data processing module 232 is set to generate imputed (eg, inferred) data to replace missing values in the drawing data 120 or the new drawing data 141. It could be. The data processing module 232 may generate (eg, determine or calculate) surrogate data using a variety of different methods. In some implementations, the imputation or inference method used by data processing module 232 to generate surrogate data is based at least in part on missing data types;

- to train model(s) 132 using drawing data 120 (including image data 122, image type 124 and other data 126) as inputs for training model(s) 132; established machine learning engine 130;

- one or more models 132 trained by the machine learning engine 130 to give a result 151 comprising a similarity value 152;

- A database 240 that stores information such as drawing data 120, new drawing data 141, result 151 (eg similarity value 152). Drawing data 120 includes image data 122 , image type 124 and other data 126 . New drawing data 141 includes image data 142 , image type 144 and other data 146 .

In some implementations, memory 206 stores metrics and/or scores determined by one or more models 132 . Memory 206 may also store thresholds and other criteria against which metrics and/or scores determined by machine learning engine 130 and/or model(s) 132 are compared. . For example, model(s) 132 may compare (eg, calculate) a similarity value to a threshold similarity value to select a similarity value that satisfies (eg, exceeds) the threshold similarity value. .

Each of the set of identified executable modules, applications or procedures described above may be stored in one or more of the above described storage devices and correspond to a set of instructions for performing the functions described above. The foregoing identified modules or programs (ie, sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of such modules may be combined or otherwise rearranged across various implementations. there is. In some implementations, memory 206 stores a subset of the identified data structures and modules described above. In addition, memory 206 may store additional modules or data structures not described above.

Although FIG. 2A illustrates a computer device 200, FIG. 2A is intended to be a functional description of various features that may be present rather than an architectural circuit diagram of an implementation described in this disclosure. Indeed, and as will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated.

2B is a schematic diagram of a server 250 according to some implementations. Server 250 may host one or more databases 290 or may provide various executable applications or modules. Server 250 generally includes one or more processing units/cores (CPUs) 252, one or more network interfaces 262, memory 264, and one or more communication buses 254 interconnecting these components. do. In some implementations, server 250 includes a user interface 256 that includes a display 258 and one or more input devices 260 such as a keyboard and mouse. In some implementations, communication bus 254 includes circuitry (sometimes referred to as a chipset) that interconnects and controls communications between system components.

In some implementations, memory 264 includes high-speed random-access memory devices, such as DRAM, SRAM, DDR RAM, or other semiconductor-based random-access solid-state memory devices, one or more magnetic disk storage devices, optical disk storage devices. devices, flash storage devices, or non-volatile solid-state storage devices using other semiconductors. In some implementations, memory 264 includes one or more storage devices located remotely from CPU(s) 252 . Memory 264 or, non-volatile storage device within memory 264 includes non-transitory computer-readable storage media.

In some implementations, memory 264 or a computer readable storage medium of memory 264 stores the following programs, modules, and data structures, or a subset thereof:

- operating system 222, which includes procedures for handling various basic system services and performing hardware dependent tasks;

- a network communication module 272 used to connect the server 250 to other computers via one or more communication networks such as the Internet, other wide area networks, local area networks, metropolitan networks, etc. and one or more communication network interfaces (wired or wireless);

- a web server 274 (such as an HTTP server) that receives web requests from users and responds by providing responsive web pages or other resources;

- a modeling application or modeling web application 280 that can be downloaded and executed by the web browser 226 on the computer device 200 of the user. In general, the modeling web application 280 has the same functionality as the desktop modeling application 230, but provides the flexibility to access from devices located in networked locations, and requires no installation and maintenance. In some implementations, modeling web application 280 includes various software modules to perform specific tasks. In some implementations, modeling web application 280 includes a graphical user interface module 282 that provides a user interface for all aspects of modeling web application 280 . In some implementations, modeling web application 280 includes drawing data 120 and new drawing data 141 as described above with respect to computer device 200;

- perform the pre-processing steps required to convert the original information into the correct data type for the drawing data 120 or the new drawing data 141, and convert the data values for the drawing data 120 or the new drawing data 141 One or more calculations based on the received raw data to generate and/or generate surrogate (e.g., inferred) data to replace missing values in drawing data 120 or new drawing data 141 as described above. Data processing module 232 for performing;

- a machine learning engine 130 for training the model(s) 132 as described above;

- one or more models 132 trained to give results 151 as described above;

- one or more databases 290 that store data used or created by the modeling web application 280 or the modeling application 230; Database 290 may store drawing data 120, new drawing data 141, results 151 (including similarity value(s) 152) and one or more models 132 as described above.

Each of the aforementioned identified executable modules, applications, or sets of procedures may be stored in one or more of the aforementioned storage devices and correspond to a set of instructions for performing the functions described above. The foregoing identified modules or programs (ie, sets of instructions) need not be implemented as separate software programs, procedures, or modules, and thus various subsets of such modules may be combined or otherwise rearranged across various implementations. there is. In some implementations, memory 264 stores a subset of the identified data structures and modules described above. In some implementations, memory 264 stores additional modules or data structures not described above.

Although FIG. 2B illustrates server 250 , FIG. 2B is intended to be a functional description of the various features that may be present rather than an architectural overview of an implementation described in this disclosure. Indeed, and as will be appreciated by those skilled in the art, items shown separately may be combined and some items may be separated. Additionally, some of the programs, functions, procedures or data shown above in relation to server 250 may be stored on or executed on computer device 200 . In some implementations, functionality and/or data may be allocated between computer device 200 and one or more servers 250. Furthermore, those skilled in the art will recognize that FIG. 2B need not represent a single physical device. In some implementations, server functionality is allocated across multiple physical devices comprising the server system. As used herein, reference to "server" includes a diverse group, collection, or array of servers that provide the described functionality, and physical servers need not physically collocate (e.g., separate Physical devices may be distributed in the United States or worldwide).

3A-3B illustrate a method of training a model (eg, a first model) of the plurality of models 132 in accordance with some implementations. To train the model of the plurality of models 132, the machine learning engine 130 uses drawing data 120 (for example, n drawings, 110-1 to 110-n). For example, training data) is received. Drawing data 120 for each drawing of plurality of drawings 110 includes image data 122 , image type 124 and, optionally, other data 126 . Machine learning engine 130 divides drawing data 120 into a first subset of drawing data 120 to be used as training data 310 and a second subset of drawing data 120 to be used as test data 312. For example, as shown in FIG. 3A , a first subset of drawing data (e.g., training data 310, drawing data 120-1 to 120-p, where p < n) is a drawing (e.g., For example, including information corresponding to the first subset of drawings 110-1 to 110-p), drawing data (eg, test data 312, drawing data 120-(p+1) to 120-n The second subset of ) includes information corresponding to the second subset of drawings (eg, drawings 110-(p+1) to 110-n). In some implementations, training data 310 (eg, a first subset of drawing data) includes at least 50%, 60%, 70%, 80%, or 90% of plurality of drawing data 120 . . For example, training data 310 (eg, a first subset of plurality of drawing data) may include 70% of plurality of drawing data 120 , and test data 312 (eg, , the second subset of the plurality of drawing data) includes 30% of the plurality of drawing data 120 .

Referring to FIG. 3B , machine learning engine 130 uses training data 310 and test data 312 to train a model of plurality of models 132 . Machine learning engine 130 trains (eg, generates) and tests in-training model 320 using training data 310 to create (eg, train) model 132-m. Data 312 is used to test and refine model 320 during training. Once model 132-m is trained, model 132-m can be used to determine the similarity of a drawing to a particular image type.

This process may be repeated for a plurality of models 132, where the drawing data 120 used as input to train each model 132 and the plurality of drawings 110 are different (e.g., separate) model.

4A-4B provide a flow diagram of a method 400 for similarity assessment between a product image and a reference drawing, in accordance with some implementations. The operation of the method 400 may be performed through a computer system corresponding to the computer device 200 or the server 250 . In some implementations, a computer system includes one or more processors and memory. 4A-4B correspond to instructions stored in a computer memory or computer readable storage medium (eg, memory 206 of computer device 200). The memory stores one or more programs configured for execution by one or more processors. For example, the operations of method 400 are performed, at least in part, by machine learning engine 130 .

According to some implementations, a computer system (eg, computer device 200, server 250, etc.) obtains (410) one or more feature vectors corresponding to one or more images of a product. A feature vector is a group of numerical features representing a particular object. For example, a computer system may acquire two images of a product (eg, a perspective view and a top view), and each image may be represented by a respective feature vector (eg, a first image of a product is represented by the first feature vector and the second image of the product represented by the second feature vector). In some implementations, the computer system receives feature vectors instead of receiving raw image files.

In some implementations, the computer system acquires ( 402 ) one or more images of the product prior to obtaining the one or more feature vectors; And extract one or more feature vectors from one or more images of the product. For example, in some implementations, a computer system receives raw images and extracts one or more feature vectors. In some implementations, a computer system receives two images and extracts a first feature vector for a first image and a second feature vector for a second image. In some implementations, the computer system includes or communicates with a camera (eg, the computer system is a cell phone with a built-in camera). A camera is used to collect images of the product, which are subsequently processed to extract one or more feature vectors.

The computer system retrieves 420 the plurality of training models built for each determination of similarity to each reference product. In some implementations, each model is set up for similarity value determination for each reference product (or each drawing of a reference product). To compare an image of a product to multiple reference products, in some implementations multiple models are used.

In some implementations, at least a first subset of the plurality of trained models is associated with each image type (eg, color photograph, black and white photograph, rendering, drawing, etc.). The computer system selects ( 422 ) one or more training models of at least a first subset of a plurality of training models associated with each image type corresponding to an image type of a particular image of the one or more images of the product. For example, if the received product image is a rendered image, the computer system selects a model set to be used with the rendered image. In another example, if the received product image is an engineering drawing, the computer system selects a model set to be used with the engineering drawing.

In some implementations, at least a second subset of the plurality of training models is not associated with a particular image type (424). For example, a deep neural network engine or open cross-domain visual search algorithm may be used to compare images of different image types.

In some implementations, each training model has been trained according to data for a plurality of products (426). The first subset of the plurality of products includes products similar to each reference product. A second subset of the plurality of products includes products that are dissimilar to each reference product. For example, each training model is trained with both positive and negative references.

For each training model of the plurality of training models, built for each similarity determination between a product and each reference product, the computer system associates each training model with one image of one or more images of the product and each reference product. It is applied to one feature vector among one or more feature vectors corresponding to one or more images of a product for determining similarity between images of a product (430, FIG. 4B). For example, as described with respect to FIG. 1B, similarity values are obtained by applying each training model to a feature vector. In some implementations, software tools such as SphereFace, CosFace, AcrFace, etc. may be used for similarity determination.

In some implementations, at least a first subset of the plurality of training models is associated with each image type. For each training model, the computer system converts one or more images of the product into a respective image of a respective image type associated with each training model (432). For example, as described with reference to FIGS. 6 and 7 , an image of a first image type may be converted into an image of a second image type different from the first image type. For conversion of images between different image types, you can use software tools such as Domain Transfer Network, Pix2Pix, etc.

In some implementations, the computer system determines 434 each similarity between the product and each reference product based on applying two or more training models built to determine the similarity between the product and each reference product. Two or more training models are associated with two or more image types.

In some implementations, applying each training model to feature vectors of the one or more feature vectors determines a distance between feature vectors of the one or more feature vectors corresponding to one or more images of the product and feature vectors extracted from images of each reference product. It includes doing (436). In some implementations, a Euclidean distance or a Cosine distance may be computed.

In some implementations, each image of the reference product includes a patent drawing (438).

The computer system provides 440 each similarity between the product and each reference product.

5A-5B provide a flow diagram of a model building method 500 for similarity assessment between a product image and a reference drawing, according to some implementations. The steps of method 500 may be performed by a computer system corresponding to computer device 200 or server 250 . In some implementations, a computer system includes one or more processors and memory. 5A-5B correspond to instructions stored in a computer memory or computer readable storage medium (eg, memory 206 of computer device 200). The memory stores one or more programs configured for execution by one or more processors.

According to some implementations, for each reference product of the plurality of reference products, a computer system (eg, computer device 200, server 250, etc.) retrieves an image of each reference product (510).

In some implementations, an image of each reference product is associated with a respective image type. The computer system selects 512 one or more of the images of each reference product corresponding to the particular image type of each image type. For example, the one or more selected images correspond to a specific image type, such as a photograph, a rendered image, an engineering drawing, or a patent drawing.

In some implementations, an image of each reference product is associated with a respective image type. The computer system converts the image of each reference product into a specific image type of each image type (514). For example, images of different image types are converted to the same image type, such as photos, rendered images, engineering drawings, or patent drawings.

For each reference product, the computer system constructs each feature vector for each image of the retrieved images of each reference product (e.g., the computer system extracts the feature vector from each image) (520). ).

The computer system determines the similarity of each reference product by training a model using at least a subset of the feature vectors (530).

In some implementations, the computer system trains the model using feature vectors associated with two or more image types (532).

In some implementations, the computer system determines 534 a respective similarity between the product and each reference product.

In some implementations, each similarity determination includes a distance determination to each feature vector (536).

In some implementations, the computer system identifies 538 one or more clusters of feature vectors, and each similarity determination includes determining a distance to each cluster of the one or more clusters.

The computer system stores the training model in a database for subsequent use in determining similarity to each reference product (540).

6 is a schematic diagram illustrating two operations involved in identifying matches in accordance with some implementations.

A first operation includes processing a received image 610 (eg, a photograph) to obtain a processed image 620 . In some implementations, the processing described above includes sensing an object sensing action. The object detection operation may be performed using software tools such as Faster Region-based Convolutional Neural Network-Region Proposal Network, Yolo, and the like. In some implementations, the processing described above includes aligning the received images (or at least partially processed images, or portions thereof). In some implementations, the processing described above includes converting the received image 610 to another type of image (eg, a photograph fit to a line drawing). The conversion operation can be performed using an algorithm such as Domain Transfer Network or Pix2Pix.

The second operation includes comparing the processed image 620 with the reference drawing 630 to determine a similarity value as described with respect to FIG. 1B.

The terminology used in the description of this disclosure is only for describing a particular implementation and is not intended to limit the disclosure. As used in this disclosure and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural as well, unless the context clearly dictates otherwise. As used herein, the term “and/or” will be understood to mean and include all possible combinations of one or more of the associated enumerated items. Also when used herein, the terms "comprises" and/or "comprising" specify the presence of specified features, steps, operations, elements and/or ingredients, but specify one or more other characteristics thereof. However, it will be understood that this does not exclude the presence or addition of steps, operations, components, components and/or groups.

[0021] [0011] [0010] [0011] [0010] [0011] [0010] [0011] [0011] [0010] [0011] [0011] In the foregoing description, for explanatory purposes, reference has been made to specific implementations. However, the foregoing illustrative discussion is not intended to be exhaustive or to limit the disclosure to the precise form disclosed. Many modifications and variations are possible in light of the above teaching. The implementations were selected and described to best explain the principles of the present disclosure and its practical applications, and accordingly, those skilled in the art can find various implementations and the present Initiation can be best used.

This work is supported by the National IT Industry Promotion Agency (NIPA) supported by the Korean government (Ministry of Science and ICT, MSIT) (A0222-21-1001) and the Seoul Business Agency (supported by the Seoul City Hall government). Seoul Commercialization Linked Technology Development (R&BD) Program (CY200002, Fake Product Warning Service through the Spectrum Analysis of the "Material Fingerprints") through Seoul Business Agency, SBA) received support from

Claims (20)

A method of evaluating similarity between a product image and a reference drawing, performed in a computing device having one or more processors and a memory storing one or more programs configured for execution by the one or more processors, the method comprising:
obtaining one or more feature vectors corresponding to one or more images of the product;
retrieving a plurality of training models built to determine each similarity of each reference product;
of the plurality of training models, for each training model, of an image of one of the one or more images of the product and the respective reference product, built to determine a respective similarity between the product and the respective reference product. applying each training model to a feature vector among the one or more feature vectors corresponding to the one or more images of the product to determine a similarity between images; and
providing the respective similarity between the product and the respective reference product. According to claim 1,
at least a first subset of the plurality of training models is associated with each image type;
The method further comprises selecting one or more training models of the at least a first subset of the plurality of training models associated with each image type corresponding to an image type of a particular image among the one or more images of the product. How to. According to claim 1,
at least a first subset of the plurality of training models is associated with each image type;
The method further comprises, for each training model, transforming the one or more images of the product into a respective image of a respective image type associated with the respective training model. According to claim 1,
wherein at least a second subset of the plurality of training models is not associated with a specific image type. According to any one of claims to 4,
Each of the above training models has been trained according to data for a plurality of products;
the first subset of the plurality of products includes products similar to each of the reference products;
wherein the second subset of the plurality of products includes products that are dissimilar to the respective reference product. 6. The method according to any one of claims to 5, wherein the method:
determining a respective similarity between the product and the respective reference product based on applying two or more training models constructed to determine a respective similarity between the product and the respective reference product;
wherein the two or more training models are associated with two or more image types. 7. The method according to any one of claims to 6, wherein the method:
acquiring the one or more images of the product; and
and extracting the one or more feature vectors from the one or more images of the product. According to any one of claims to 7,
The step of applying each training model to a feature vector of the one or more feature vectors may include extracting from the feature vector of the one or more feature vectors corresponding to the one or more images of the product and the image of each reference product. A method comprising determining a distance between feature vectors. According to any one of claims to 8,
wherein the image of each reference product comprises a patent drawing. A non-transitory computer-readable storage medium storing one or more programs configured for execution by a computer system having one or more processors and memory,
A non-transitory computer-readable storage medium, wherein the one or more programs include instructions for performing any one of the methods of claims 1 to 9. As a computer system:
one or more processors;
Memory; and
one or more programs stored in the memory and configured for execution by the one or more processors;
A computer system, wherein the one or more programs include instructions for performing any one of the methods of claims 1 to 9. A method of building a model for evaluating similarity between a product image and a reference drawing, performed in a computing device having one or more processors and a memory storing one or more programs configured for execution by the one or more processors, the method comprising: :
For each reference product in the plurality of reference products:
Retrieving an image of each reference product;
constructing each feature vector for each image of the retrieved image of each reference product;
using at least a subset of feature vectors to train a model that determines the similarity of each reference product; and
storing the trained model in a database for subsequent use in determining the similarity of each reference product. According to claim 12,
the image of each reference product is associated with a respective image type;
The method further comprises selecting one or more images of the images of each reference product corresponding to a particular image type of the respective image type. According to claim 12,
the image of each reference product is associated with a respective image type;
The method further comprises converting the image of the respective reference product into an image of a specific image type of the respective image type. According to claim 12,
training the model using feature vectors associated with two or more image types. According to any one of claims 12 to 15,
determining a respective similarity between a product and the respective reference product. According to claim 16,
Wherein determining each similarity comprises determining a distance to each feature vector. The method of claim 16 or 17,
further comprising identifying one or more clusters of the feature vectors;
wherein determining each similarity comprises determining a distance to each cluster of the one or more clusters. A non-transitory computer-readable storage medium storing one or more programs configured for execution by a computer system having one or more processors and memory,
A non-transitory computer-readable storage medium, wherein the one or more programs include instructions for performing any one of the methods of claims 12-18. As a computer system:
one or more processors;
Memory; and
one or more programs stored in the memory and configured for execution by the one or more processors;
A computer system, wherein the one or more programs include instructions for performing any one of the methods of claims 12 to 18.

Images