KR20230101673A - Method and electronic device for recognizing product - Google Patents

Abstract

First feature information and second feature information are obtained from an image related to a product, and a fusion based on the first feature information and the second feature information is performed using a main encoder model that reflects a correlation between feature information of different modalities. Disclosed are a method and an electronic device for obtaining feature information, performing matching with fusion feature information using a product database, and recognizing a product providing product information according to a matching result.

Description

Product recognition method and electronic device {METHOD AND ELECTRONIC DEVICE FOR RECOGNIZING PRODUCT}

It relates to a product recognition method and an electronic device.

A unique identification code such as a barcode or QR code is attached to the product to classify the product. Because consumers cannot analyze this unique identification code without a device such as a reader, a label is attached to the product for consumers.

Recently, with the development of mobile devices such as smart phones, interest in a method for consumers to search for products or identify products using smart phones is growing.

An object of the present invention is to provide a method and an electronic device for recognizing a product based on fusion feature information reflecting a correlation between feature information of different modalities.

According to one aspect of the present disclosure, a method for recognizing a product includes obtaining first feature information and second feature information from an image related to the product, and a main encoder model reflecting a correlation between feature information of different modalities. Obtaining fusion feature information based on the first feature information and the second feature information using , performing matching with the fusion feature information using the product database, and and providing information on the product according to a result of the matching.

According to one aspect of the present disclosure, an electronic device for recognizing a product includes a display, a memory storing one or more instructions, and a processor executing the one or more instructions stored in the memory, wherein the processor executes the one or more instructions. By executing, first feature information and second feature information are obtained from an image related to a product, and the first feature information and the second feature information are obtained by using a main encoder model that reflects a correlation between feature information of different modalities. 2 Acquire fusion feature information based on the feature information, perform matching with the fusion feature information using the database of the product, and provide information on the product through the display according to a result of the matching.

According to one aspect of the present disclosure, a computer readable recording medium having a program for executing the above method may be provided.

1 is a diagram illustrating a process of recognizing a product by an electronic device according to an embodiment of the present disclosure.
2 is a diagram for explaining a feature information encoder model used in a process of recognizing a product according to an embodiment of the present disclosure.
3 is a flowchart illustrating a method of recognizing a product according to an embodiment of the present disclosure.
4 is a flowchart illustrating a process of acquiring image feature information and text feature information according to an embodiment of the present disclosure.
5 is a diagram for explaining a main encoder model according to an embodiment of the present disclosure.
6 is a flowchart illustrating a process of performing matching with one or more feature information including fusion feature information using a product database according to an embodiment of the present disclosure.
7 is a diagram for explaining an example of matching based on text feature information.
8 is a diagram for explaining an example of matching based on image feature information.
9 is a diagram illustrating a case in which a product recognition process according to an embodiment of the present disclosure is operated in an electronic device.
10 is a diagram illustrating a case in which a product recognition process according to an embodiment of the present disclosure is operated based on an electronic device and a server.
11 is a block diagram illustrating the configuration of an electronic device according to an embodiment of the present disclosure.
12 is a block diagram showing the configuration of a server according to an embodiment of the present disclosure.

Terms used in this specification will be briefly described, and the present disclosure will be described in detail. In this disclosure, the expression “at least one of a, b, or c” means “a”, “b”, “c”, “a and b”, “a and c”, “b and c”, “a, b” and c”, or variations thereof.

The terms used in the present disclosure have been selected from general terms that are currently widely used as much as possible while considering the functions in the present disclosure, but they may vary according to the intention or precedent of a person skilled in the art, the emergence of new technologies, and the like. In addition, in a specific case, there are also terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the corresponding description. Therefore, terms used in the present disclosure should be defined based on the meaning of the term and the general content of the present disclosure, not simply the name of the term.

Singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art described herein. Also, terms including ordinal numbers such as 'first' or 'second' used in this specification may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another.

When it is said that a certain part "includes" a certain component throughout the specification, it means that it may further include other components without excluding other components unless otherwise stated. In addition, terms such as “unit” and “module” described in the specification refer to a unit that processes at least one function or operation, and may be implemented as hardware or software or a combination of hardware and software.

Hereinafter, with reference to the accompanying drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily carry out the present invention. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments described herein.

Hereinafter, the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a process of recognizing a product by an electronic device 100 according to an embodiment of the present disclosure.

In the present disclosure, the electronic device 100 may include a smart phone, smart glasses, a wearable device, a personal digital assistant (PDA) or a portable terminal device such as a mobile scanner, but is not limited thereto. The electronic device 100 may be implemented with various types and types of devices including an input unit for obtaining an image and an output unit for providing information.

In the present disclosure, a product refers to any object that is bought and sold. For example, products may be daily necessities, industrial products, electronic products, and the like.

A user of the electronic device 100, for example, a consumer who wants to purchase a product may use the electronic device 100 to obtain information about the product. A user may directly input information on a label attached to a product into the electronic device 100 to search for information about the product. As a simpler method, the user may take a picture of a label attached to a product by using a camera provided in the electronic device 100 . The electronic device 100 may obtain product information by executing a predetermined algorithm based on the captured label image.

Referring to FIG. 1 , a user of the electronic device 100 uses a camera function of the electronic device 100 to take a picture of a wine label in order to obtain information about wine. Although wine is shown as a product in FIG. 1 , the product is not limited thereto, and for convenience of description, a case in which the product is wine will be described below.

The electronic device 100 may perform a process for recognizing a product based on a photographed image related to the product, that is, an image of a label. To this end, the electronic device 100 may store the feature information encoder model in memory.

The feature information encoder model may output a plurality of feature information from an image related to wine. The feature information encoder model may be composed of one or more models that encode and output feature information, and each model may be in the form of a neural network composed of a plurality of layers.

For example, the feature information encoder model shown in FIG. 1 may output at least one piece of feature information for recognizing wine based on an image included in a wine label or text such as letters, numbers, and symbols.

The electronic device 100 may recognize wine based on at least one piece of feature information acquired using the feature information encoder model, and provide information on the recognized wine to the user. For example, the electronic device 100 may perform matching with at least one piece of feature information using a database in which information on wine is stored, and may provide information on wine according to a matching result.

2 is a diagram for explaining a feature information encoder model used in a process of recognizing a product according to an embodiment of the present disclosure.

The feature information encoder model may encode and output feature information from an image related to a product. The feature information encoder model may include each encoder model corresponding to feature information to be encoded.

Referring to FIG. 2 , a feature information encoder model may include a first sub encoder model, a second sub encoder model, and a main encoder model. However, unlike shown in FIG. 2, the type or number of detailed configuration models of the feature information encoder model may be changed according to the type or number of feature information used for product recognition.

The first sub-encoder model and the second sub-encoder model take an image related to a product as an input, but respectively output characteristic information of different modalities. An image related to a product may be classified into a first element and a second element according to the types of elements constituting the image. For example, the first element may be an image element and the second element may be a text element. An image related to a product may be classified into an image element and a text element in consideration of attributes of an image area and a text area.

The first sub-encoder model may extract a first feature from the first element classified in the image and encode the extracted first feature as first feature information. The first sub-encoder model may be in the form of a neural network including a plurality of layers. For example, the first characteristic information may be image characteristic information. The image feature information may be information indicating a characteristic of a form such as shape, shape, color, etc. recognized from an image.

The second sub-encoder model may extract a second feature from the second element classified in the image and encode the extracted second feature as second feature information. The second sub-encoder model may be in the form of a neural network including a plurality of layers. For example, the second characteristic information may be text characteristic information. The text characteristic information may be information indicating characteristics related to the meaning of letters, numbers, symbols, etc. recognized from the text.

The first feature information, which is an output of the first sub-encoder model, and the second feature information, which is an output of the second sub-encoder model, may be input to the main encoder model. The main encoder model may receive at least one piece of first feature information and at least one piece of second feature information. For example, m pieces of first feature information and n pieces of second feature information may be input to the main encoder model. Unlike shown in FIG. 2 , at least one piece of third feature information having a modality different from that of the first feature information and the second feature information may be input.

The main encoder model may encode fusion feature information by reflecting a correlation between feature information of different modalities. Fusion feature information is feature information newly created from feature information of different modalities, and may be fusion of first feature information and second feature information, but is not limited thereto, and feature information of another modality is further fused. it could be For example, new feature information may be encoded by fusion of image feature information and text feature information.

The main encoder model is based on self-attention, and for any one feature information, it finds out the correlation with all the feature information input to the main encoder model, and any one feature information in the image related to the product. An attention value of can be obtained. Correlation means relevance or importance between feature information. The main encoder model may obtain an attention value of any one feature information by obtaining a weighted sum reflecting the identified correlation. The main encoder model may obtain each attention value for all feature information input to the main encoder model. Each attention value may be a context vector in which importance or characteristics of each feature information in relation to other feature information are considered. The main encoder model may output fusion feature information of an image related to a product by collecting attention values of all feature information.

As a result, the feature information encoder model obtains first feature information and second feature information from the image related to the product, and obtains fusion feature information based on the first feature information and the second feature information using the main encoder model. can The fusion feature information has a value in which the importance and characteristics of at least one piece of first feature information and at least one piece of second feature information are reflected in an image related to a product. The value of the fusion feature information is classified through matching with a product database, and can be used to recognize the product.

Hereinafter, for convenience of description, a case in which the first characteristic information is image characteristic information and the second characteristic information is text characteristic information will be described as an example, but is not limited thereto. Specific operations for recognizing products will be described in more detail in the drawings and detailed descriptions thereof to be described later.

3 is a flowchart illustrating a method of recognizing a product according to an embodiment of the present disclosure.

In step 310, the electronic device 100 may obtain image characteristic information and text characteristic information from an image related to a product. An image related to a product may be obtained through a camera of the electronic device 100 by executing a product recognition application according to a user's input. Step 310 will be described in detail in FIG. 4 below.

4 is a flowchart illustrating a process of acquiring image feature information and text feature information according to an embodiment of the present disclosure.

The electronic device 100 may classify images related to products into image elements and text elements. Depending on the properties of each element constituting an image, it can be divided into an image element and a text element. For example, the electronic device 100 may detect the image area within the image by checking the edges of objects in the image related to the product. The electronic device 100 may execute an Optical Character Recognition (OCR) function to detect a text area in an image. At least one detected image area may be classified as an image element, and at least one text area may be classified as a text element.

If the product is wine, at least one image area related to wine may be included in the wine label. For example, a pattern of a winery that produced wine, a symbol pattern of a grape shape corresponding to a raw material of wine, and the like may be included in the label. In addition, at least one text area related to wine may be included in the wine label. For example, the name of the wine, vintage year of the wine, country of origin of the wine, etc. may be included in the label.

The electronic device 100 may extract image features from image elements classified in the image using the first sub-encoder model, and encode the extracted image features as image feature information. The first sub-encoder model is a neural network including a plurality of layers, and image elements may be input to an input layer and image feature information may be output from an output layer. When a plurality of distinct image regions are detected in an image related to a product, the first sub-encoder model extracts a plurality of image features, encodes each image feature into image feature information, and outputs a plurality of image feature information. can

For example, the image feature information may be expressed as a vector encoding a feature about a shape of an image corresponding to an image region included in a wine label. When a plurality of image features are extracted from the wine label, the first sub encoder model may output each of the plurality of vectors as image feature information.

The electronic device 100 may extract text features from text elements classified in the image using the second sub-encoder model, and encode the extracted text features into text feature information. The second sub-encoder model is a neural network including a plurality of layers, and text elements may be input to the input layer and text feature information may be output from the output layer. When a plurality of distinct text regions are detected in an image related to a product, the second sub-encoder model extracts a plurality of text features, encodes each text feature into text feature information, and outputs a plurality of text feature information. can

For example, the text feature information may be a vector encoding a feature related to the meaning of text corresponding to a text area included in a wine label. When a plurality of text features are extracted from the wine label, the second sub encoder model may output each of the plurality of vectors as text feature information.

Referring back to FIG. 3 , in step 320, the electronic device 100 obtains fusion feature information based on image feature information and text feature information by using a main encoder model that reflects the correlation between different feature information. can Step 320 will be described in detail with reference to FIG. 5 below.

5 is a diagram for explaining a main encoder model according to an embodiment of the present disclosure.

The main encoder model may receive at least one image feature information and at least one text feature information as inputs. Referring to FIG. 5 , 1 piece of image feature information and n-1 pieces of text feature information may be input to the input layer of the main encoder model. One vector corresponding to the image feature information output from the first sub-encoder model described above and n-1 vectors corresponding to text feature information output from the second sub-encoder model may be input to the input layer of the main encoder model. there is.

For example, as shown in FIG. 5 , one image feature corresponding to one image region is extracted and encoded from the wine label, and one image feature information may be input to the input layer of the main encoder model. there is. In addition, three text areas are detected from the wine label, and by using the OCR function, three text features corresponding to the name of the wine 'MONTES', the vintage year '2001', and the wine's country of origin 'Chile' After being extracted and encoded, the three text feature information can be input to the input layer of the main encoder model.

The main encoder model is based on self-attention, and for any one feature information, it identifies the correlation with all feature information input to the main encoder model, and the attention value (attention value) of any one feature information in the image related to the product. value) can be obtained. The main encoder model correlates any one feature information with all input feature information through a matrix multiplication operation between the query vector extracted from any one feature information and the key vectors extracted from all input feature information. relationship can be identified. At this time, the correlation means the relevance or importance between feature information. The main encoder model may obtain an attention value of any one feature information by obtaining a weighted sum reflecting the identified correlation. The main encoder model may obtain each attention value for all feature information input to the main encoder model.

For example, referring to FIG. 5 , all feature information input to the input layer through a vector extraction layer including at least one layer connected to the input layer of the main encoder model and a calculation layer including at least one layer. For each of , a context vector corresponding to the attention value may be obtained.

A first query vector q1 and a first key vector k1 may be extracted from the first vector, which is image feature information input to the input layer of the main encoder model. A second query vector q2 and a second key vector k2 may be extracted from the second vector, which is text feature information corresponding to 'MONTES' input to the input layer of the main encoder model. A third query vector q3 and a third key vector k3 may be extracted from the third vector, which is text feature information corresponding to '2001' input to the input layer of the main encoder model. An n-th query vector (qn) and an n-th key vector (kn) may be extracted from the n-th vector, which is text feature information corresponding to 'Chile' input to the input layer of the main encoder model.

Then, for each of all the specific information input to the input layer of the main encoder model, a context vector corresponding to the attention value may be obtained in the following manner.

First, in order to determine the correlation between the first vector, which is image feature information, and all feature information, the matrix multiplication of the first query vector (q1) and the first key vector (k1) to the n-th key vector (kn), respectively. Through the operation, the dot product values q1k1, q1k2, q1k3, ..., q1kn can be obtained. The higher the correlation with the image feature information, which is the first vector, the higher the dot product value, and contextually, the higher the correlation between them. By using the weight value obtained by performing the softmax operation on each dot product value and obtaining the weighted sum of the value vector of each feature information, the meaning of the image feature information in the image related to the product, that is, the wine label, is converted into the context vector V1. can express

Next, in order to determine the correlation between the second vector, which is the first text characteristic information corresponding to 'MONTES', and all the characteristic information, the second query vector (q2) and the first key vector (k1) to the nth key Dot product values q2k1, q2k2, q2k3, ..., q2kn can be obtained through matrix multiplication of each vector kn. The meaning of the first text feature information corresponding to 'MONTES' in the wine label is obtained by obtaining a weighted sum of the value vector of each feature information using a weight value obtained by performing a softmax operation on each dot product value. It can be expressed as V2.

In a similar way, in order to determine the correlation between the third vector, which is the second text feature information corresponding to '2001', and all feature information, the third query vector (q3) and the first key vectors (k1 to th) Dot product values q3k1, q3k2, q3k3, ..., q3kn can be obtained through matrix multiplication of n key vectors (kn). The meaning of the second text feature information corresponding to '2001' in the label of wine is obtained by obtaining a weighted sum of the value vector of each feature information using a weight value obtained by performing a softmax operation on each dot product value. It can be expressed as V3.

Finally, in order to determine the correlation between the n-th vector, which is the n-1th text feature information corresponding to 'Chile', and all feature information, the n-th query vector (qn) and the first key vector (k1 to Dot product values qnk1, qnk2, qnk3, ..., qnkn can be obtained through matrix multiplication of each nth key vector (kn). The meaning of the n-1th text feature information corresponding to 'Chile' in the wine label is obtained by obtaining the weighted sum of the value vector of each feature information using the weight value obtained by performing the softmax operation on each dot product value. It can be expressed as a context vector Vn.

Each attention value in the form of a context vector may be a value reflecting a contextual meaning of each characteristic information in relation to other characteristic information in an image related to a product.

The main encoder model may output the Z1 vector, which is fusion feature information, by summating attention values of all feature information, that is, context vectors V1, V2, V3, ..., Vn. As a result, the fusion feature information can be a new type of feature information in which the meanings of at least one image feature information and at least one piece of text feature information are reflected in an image related to a product. That is, from an image related to a product, a new type of feature information in which context is understood by utilizing both morphological feature information and semantic feature information can be encoded.

Referring back to FIG. 3 , in step 330, the electronic device 100 may perform matching with fusion feature information using a product database. Step 330 will be described in detail with reference to FIG. 6 below.

6 is a flowchart illustrating a process of performing matching with one or more feature information including fusion feature information using a product database according to an embodiment of the present disclosure.

In step 610, the electronic device 100 may check whether the fusion feature information matches the product database. The electronic device 100 may compare the fusion feature information with the information registered in the product database and determine whether there is a match.

If the fusion feature information does not match the product database, the electronic device 100 may match the image feature information and the text feature information with the product database.

If the fusion feature information does not match the product database, in step 620, the electronic device 100 may first check whether the text feature information matches the product database. When there is a plurality of pieces of text feature information, matching may be performed with a product database for all of the pieces of text feature information.

If the text feature information does not match the product database, in step 630, the electronic device 100 may check whether the image feature information matches the product database. When there is a plurality of image feature information, matching may be performed with a product database for all of the plurality of image feature information.

7 is a diagram for explaining an example of matching based on text feature information.

Referring to FIG. 7 , it shows a case where the design of the label attached to the wine is changed. The two wines are the same product, but they are attached to labels with different designs. One of the two wines is the wine the consumer wants to purchase, and the other is the wine registered in the product database.

In this case, the image feature information encoded from the labels of the two wines is different. However, the text feature information encoded from the labels of the two wines may be the same. As shown in FIG. 7, since 'Gran Vino 2001' is encoded as text feature information for both wines, the text feature information of the wine that the consumer wants to purchase can be matched with information stored in the product database. .

8 is a diagram for explaining an example of matching based on image feature information.

Referring to Figure 8, it shows a case where the label attached to the wine is discolored or damaged. The two wines are the same product, but the wine that the consumer wants to purchase has a discolored or damaged label attached during the distribution process.

In particular, when the discolored or damaged portion is the text area of the label, the text feature information encoded from the respective labels of the two wines is different. However, the image feature information encoded from the respective labels of the two wines may be identical to each other. As shown in FIG. 8, in the first wine, 'Gran vino 2001' was encoded as text feature information, and in the second wine, 'Gran vino' part was discolored, and as a result of OCR execution, only '2001' part was text feature information. can be encoded as As a result, text feature information of the two wines may not match. However, since the characteristics of the shape of the image area of the respective labels of the two wines are almost the same, the image characteristic information encoded from the respective labels of the two wines can be matched with each other. As shown in FIG. 8 , image characteristic information of wine that a consumer intends to purchase may be matched with information stored in a product database.

When the text feature information and the image feature information do not match the product database, in step 640, the electronic device 100 may update the database by registering the corresponding product as a new product in the product database.

When any one of the image feature information and the text feature information matches the product database, in step 650, the electronic device 100 converts the fusion feature information that does not match the product database in step 610 to the product database later. The main encoder model may be updated to match.

For example, fusion feature information may be distorted due to partial damage or contamination of an image related to a product, OCR misrecognition, and the like. In this case, fusion feature information is not matched to the product database, but text feature information or image feature information encoded from a problem-free region in the image related to the product may be matched to the product database. Unmatched fusion feature information may be a sample requiring continuous learning. If the main encoder model is updated according to continuous learning, it may be possible to match with the product database for similarly distorted fusion feature information in the future.

Referring back to FIG. 6 , in step 660, the electronic device 100 may return a matching result when any one of fusion feature information, text feature information, and image feature information matches the product database. The electronic device 100 may retrieve matched product information from a product database.

Referring back to FIG. 3 , in step 340, the electronic device 100 may provide product information according to a matching result. When there is a matched product from the product database, the electronic device 100 may provide information on the matched product to the display. When there is no matched product from the product database, the electronic device 100 may notify that the product does not exist in the product database and inform that information on the corresponding product is updated in the product database.

9 is a diagram illustrating a case in which a product recognition process according to an embodiment of the present disclosure is operated in the electronic device 100 .

Server 200 may be composed of at least one function server. The first server may train a feature information encoder model, in particular, a main encoder model. When a sample requiring learning is generated, the first server may learn the main encoder model using the corresponding sample. The second server may manage a product database. When information related to a product is changed or registration of a new product is required, the second server may update the database of the product with the new information.

The electronic device 100 may receive the main encoder model learned in the server 200 and update the main encoder model stored in the electronic device 100 . The electronic device 100 may receive a database of products from the server 200 and maintain the database stored in the electronic device 100 as up-to-date. The electronic device 100 may receive the main encoder model and the product database from the server 200 whenever the product recognition application is executed or periodically.

In this way, when the electronic device 100 receives the feature information encoder model including the main encoder model and the product database from the server 200 and stores them in the electronic device 100, the electronic device 100 provides an on-device (On-device) based product recognition process can be operated.

Referring to FIG. 9 , the electronic device 100 may obtain an image related to wine, that is, a wine label, through a camera, by executing a wine recognition application according to a user's input. The electronic device 100 may obtain image feature information and text feature information from the wine label by using the first sub-encoder model and the second sub-encoder model of the feature information encoder model. The electronic device 100 may obtain fusion feature information based on image feature information and text feature information by using the main encoder model of the feature information encoder model. The electronic device 100 may perform matching with at least one piece of feature information including fusion feature information by using the wine database. The electronic device 100 may provide wine information to the display according to the matching result.

10 is a diagram illustrating a case in which a product recognition process according to an embodiment of the present disclosure is operated based on the electronic device 100 and the server 200.

In the above, for convenience of description, the subject of the product recognition method has been described as the electronic device 100, but the product recognition process can also be performed in the server 200 associated with the electronic device 100. The information about the method of recognizing a product by the electronic device 100 described above may also be applied to the embodiment of FIG. 10 .

Referring to FIG. 10 , the server 200 may learn a feature information encoder model, in particular, a main encoder model, and manage a product database. The electronic device 100 may not receive a main encoder model or a product database from the server 200 , but instead may transmit an image related to a product acquired by the electronic device 100 to the server 200 . As such, when the feature information encoder model including the main encoder model and the product database are not stored in the electronic device 100, the product recognition process can be operated based on the electronic device 100 and the server 200. .

Referring to FIG. 10 , the electronic device 100 may transmit an image related to wine acquired through a camera, that is, a wine label to the server 200 . The server 200 may obtain image feature information and text feature information from the wine label by using the first sub-encoder model and the second sub-encoder model of the feature information encoder model. The server 200 may obtain fusion feature information based on image feature information and text feature information by using the main encoder model of the feature information encoder model. The server 200 may perform matching with at least one piece of feature information including fusion feature information by using a wine database. The server 200 may transmit a matching result to the electronic device 100 . The electronic device 100 may provide wine information to the display according to the received matching result.

11 is a block diagram showing the configuration of an electronic device 100 according to an embodiment of the present disclosure.

Referring to FIG. 11 , an electronic device 100 according to an embodiment may include a memory 110, a processor 120, a camera 130, a display 140, and a communication interface 150.

The memory 110 may store instructions, data structures, and program codes readable by the processor 120 . In the disclosed embodiments, operations performed by the processor 120 may be implemented by executing instructions or codes of a program stored in the memory 110 .

The memory 110 includes a flash memory type, a hard disk type, a multimedia card micro type, and a card type memory (eg SD or XD memory). Non-volatile memory including at least one of ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk, and optical disk and volatile memory such as RAM (Random Access Memory) or SRAM (Static Random Access Memory).

The memory 110 according to an embodiment may store one or more instructions and/or programs for controlling the electronic device 100 to recognize a product. For example, an encoder module and a product matching module may be stored in the memory 110 . If learning of an artificial intelligence model is required within the electronic device 100, a model learning module may be further installed.

The processor 120 may control operations or functions performed by the electronic device 100 by executing instructions stored in the memory 110 or programmed software modules. The processor 120 may be composed of hardware components that perform arithmetic, logic and input/output operations and signal processing. The processor 120 may control overall operations for the electronic device 100 to recognize products by executing one or more instructions stored in the memory 110 .

The processor 120 may include, for example, a central processing unit, a microprocessor, a graphic processing unit, application specific integrated circuits (ASICs), digital signal processors (DSPs), DSPDs ( Digital Signal Processing Devices), PLDs (Programmable Logic Devices), FPGAs (Field Programmable Gate Arrays), Application Processors, Neural Processing Units, or artificial intelligence designed with hardware structures specialized in processing AI models. It may be configured with at least one of dedicated processors, but is not limited thereto. Each processor constituting the processor 120 may be a dedicated processor for performing a predetermined function.

The camera 130 is a hardware module that acquires images. The camera 130 may capture images related to products. The camera 130 may include at least one camera module, and may support macro, depth-of-field, telephoto, wide-angle, and ultra-wide-angle functions according to specifications of the electronic device 100 .

The display 140 includes an output unit for providing information or images, and may further include an input unit for receiving an input. The output unit may include a display panel and a controller controlling the display panel, and may include a variety of display devices such as OLED (Organic Light Emitting Diodes) displays, AM-OLED (Active-Matrix Organic Light-Emitting Diodes) displays, LCD (Liquid Crystal Display) displays, and the like. can be implemented in this way. The input unit may receive various types of inputs from a user, and may include at least one of a touch panel, a keypad, and a pen recognition panel. The display 140 may be provided in the form of a touch screen in which a display panel and a touch panel are combined, and may be implemented in a flexible or foldable manner.

The communication interface 150 may perform wired/wireless communication with other devices or networks. The communication interface 150 may include a communication circuit or communication module supporting at least one of various wired/wireless communication methods. For example, the communication interface 150 may be, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), infrared Communication (IrDA, Infrared Data Association), BLE (Bluetooth Low Energy), NFC (Near Field Communication), WiBro (Wireless Broadband Internet, Wibro), WiMAX (World Interoperability for Microwave Access, WiMAX), SWAP (Shared Wireless Access Protocol) Data communication between the electronic device 100 and other devices may be performed using at least one of data communication methods including WiGig, Wireless Gigabit Alliances (RF), and RF communication.

The communication interface 150 according to an embodiment may transmit/receive an artificial intelligence model (eg, a CNN-based feature information encoding model, a main encoding model) used to recognize a product or sample data for learning with an external device. . The communication interface 150 may transmit an image captured through the camera 130 of the electronic device 100 to the server 200 or may receive an artificial intelligence model or database learned in the server 200 .

The processor 120 or a first processor constituting the processor 120 may execute a product recognition application and control the electronic device 100 to obtain an image related to the product. An image related to a product may be acquired using the camera 130 of the electronic device 100 or received from an external device.

The processor 120 or a second processor constituting the processor 120 executes an encoder module to obtain first feature information and second feature information from an image related to a product, and to obtain the first feature information and the second feature information. Based fusion feature information may be obtained.

The processor 120 or the second processor executes an encoder module to classify an image related to a product into a first element and a second element, and encodes first characteristic information from the first element using a first sub-encoder model. And, the second feature information may be encoded from the second element by using the second sub-encoder model. The processor 120 or the second processor executes an encoder module to extract a first feature from a first element classified in an image related to a product using a first sub-encoder model, and converts the extracted first feature into a first sub-encoder model. It can be encoded as feature information. The processor 120 or the second processor executes the encoder module to extract a second feature from the second element classified in the image related to the product by using the second sub-encoder model, and converts the extracted second feature to the second element. It can be encoded as feature information.

The processor 120 or the second processor executes the encoder module to generate fusion feature information based on the first feature information and the second feature information using a main encoder model that reflects the correlation between feature information of different modalities. can be obtained

The processor 120 or a third processor constituting the processor 120 may execute a product matching module and perform matching with fusion feature information using a database of products. The processor 120 or the third processor may execute the product matching module and, if the fusion feature information does not match, match the first feature information and the second feature information with a product database. The processor 120 or the third processor may provide product information through the display 140 according to a matching result.

The processor 120 or the fourth processor constituting the processor 120 executes the model learning module when the fusion feature information is not matched and either the first feature information or the second feature information matches the product database. Thus, the main encoder model may be updated so that unmatched fusion feature information is matched with the product database. The processor 120 or the fourth processor executes the model learning module to learn the main encoder model using unmatched fusion feature information, or to use the unmatched fusion feature information to learn the main encoder model as a server ( 200) and can be controlled to be updated.

If none of the fusion feature information, the first feature information, and the second feature information match the product database, the processor 120 or the fifth processor constituting the processor 120 converts the product as a new product. can be registered in the database.

12 is a block diagram showing the configuration of a server 200 according to an embodiment of the present disclosure.

Operations of the electronic device 100 described above may be performed in a similar manner in the server 200 . The server 200 according to an embodiment may include a memory 210 , a processor 220 , a communication interface 230 , and a storage 240 . Components of the server 200 may correspond to the memory 110, the processor 120, and the communication interface 130 of the electronic device 100 of FIG. 11, respectively, and thus the same description as the previous description will be omitted.

The memory 210 may store various data, programs, or applications for driving and controlling the server 200 . One or more instructions or applications stored in memory 210 may be executed by processor 220 . A module performing the same function as a module stored in the electronic device 100 may be stored in the memory 210 . For example, the memory 210 may store an encoder module, a product matching module, a model learning module, and data and program command codes corresponding thereto. The processor 220 may control the server 200 as a whole. The processor 220 according to an embodiment may execute one or more instructions stored in the memory 210 .

The communication interface 230 includes a local area network (LAN), a wide area network (WAN), a value added network (VAN), a mobile radio communication network, a satellite communication network, and these It may include one or more components that allow communication through mutual combination.

The storage 240 may store feature information encoding models including a first sub encoding model, a second sub encoding model, and a main encoding model. The storage 240 may store training datasets used to learn various types of artificial intelligence models. The storage 240 may store a product database that stores product-related information.

The server 200 according to an embodiment may be a device with higher computing performance than the electronic device 100 so as to be able to perform more calculations faster than the electronic device 100 . The server 200 may perform learning of an artificial intelligence model, which requires a relatively large amount of computation.

Meanwhile, the embodiments of the present disclosure may be implemented in the form of a recording medium including instructions executable by a computer, such as program modules executed by a computer. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include computer storage media and communication media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Communication media may typically include computer readable instructions, data structures, or other data in a modulated data signal such as program modules.

Also, the computer-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary storage medium' only means that it is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is stored semi-permanently in the storage medium and temporary It does not discriminate if it is saved as . For example, a 'non-temporary storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided by being included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store or between two user devices (eg smartphones). It can be distributed (e.g., downloaded or uploaded) directly or online. In the case of online distribution, at least a part of a computer program product (eg, a downloadable app) is stored on a device-readable storage medium such as a memory of a manufacturer's server, an application store server, or a relay server. It can be temporarily stored or created temporarily.

The description of the present disclosure described above is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present disclosure. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present disclosure is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be interpreted as being included in the scope of the present disclosure. do.

Claims (20)

In the method of recognizing a product,
obtaining first characteristic information and second characteristic information from an image associated with a product;
obtaining fusion feature information based on the first feature information and the second feature information by using a main encoder model reflecting a correlation between feature information of different modalities;
performing matching with the fusion feature information using the product database; and
and providing information of the product according to a result of the matching. According to claim 1,
The main encoder model,
at least one of the first characteristic information and at least one of the second characteristic information as input;
Obtaining an attention value of any one feature information in the image by finding a correlation with all of the input feature information for any one feature information based on self-attention;
The method of outputting the fusion feature information by summation of attention values of all input feature information. According to claim 2,
The main encoder model,
Through a matrix multiplication operation between a query vector extracted from any one of the feature information and key vectors extracted from all of the input feature information, the correlation between the one feature information and all of the input feature information A method of acquiring the attention value by identifying a relationship and obtaining a weighted sum reflecting the identified correlation. According to claim 1,
The first characteristic information is image characteristic information, and the second characteristic information is text characteristic information. According to claim 1,
Obtaining the first characteristic information and the second characteristic information,
classifying the image into a first element and a second element;
extracting a first feature from a first element classified in the image using a first sub-encoder model, and encoding the extracted first feature as the first feature information; and
and extracting a second feature from a second element classified in the image by using a second sub-encoder model, and encoding the extracted second feature as the second feature information. According to claim 1,
If the fusion feature information does not match, the method further comprising performing matching with a database of the product on the first feature information and the second feature information. According to claim 5,
If any one of the first feature information and the second feature information matches the database of the product, updating the main encoder model so that the unmatched fusion feature information matches the database of the product. Including, how. According to claim 5,
The method further comprising registering the product as a new product in the database of the product when the first characteristic information and the second characteristic information do not match in the database of the product. According to claim 1,
The method further comprising receiving the main encoder model and the database of the product from at least one server that trains the main encoder model and manages the database of the product. According to claim 1,
According to a user's input, executing a product recognition application, further comprising acquiring an image related to the product through a camera, the method. In an electronic device for recognizing a product,
display;
a memory that stores one or more instructions; and
a processor to execute the one or more instructions stored in the memory;
By executing the one or more instructions, the processor:
First feature information and second feature information are obtained from an image related to a product, and by using a main encoder model that reflects a correlation between feature information of different modalities, the first feature information and the second feature information An electronic device that obtains based fusion feature information, performs matching with the fusion feature information using a database of the product, and provides information of the product through the display according to a result of the matching. According to claim 11,
The main encoder model,
at least one of the first characteristic information and at least one of the second characteristic information as input;
Obtaining an attention value of any one feature information in the image by finding a correlation with all of the input feature information for any one feature information based on self-attention;
The electronic device outputting the fusion feature information by collecting attention values of all input feature information. According to claim 12,
The main encoder model,
Through a matrix multiplication operation between a query vector extracted from any one of the feature information and key vectors extracted from all of the input feature information, the correlation between the one feature information and all of the input feature information An electronic device that obtains the attention value by identifying a relationship and obtaining a weighted sum reflecting the identified correlation. According to claim 11,
By executing the one or more instructions, the processor:
classifying the image into a first element and a second element;
Extracting a first feature from a first element classified in the image using a first sub-encoder model, encoding the extracted first feature as the first feature information,
The electronic device of claim 1 , wherein a second feature is extracted from a second element classified in the image by using a second sub-encoder model, and the extracted second feature is encoded as the second feature information. According to claim 11,
By executing the one or more instructions, the processor:
If the fusion feature information does not match, matching the first feature information and the second feature information with a database of the product is performed. According to claim 15,
By executing the one or more instructions, the processor:
When any one of the first feature information and the second feature information matches the database of the product, the electronic device updates the main encoder model so that the unmatched fusion feature information matches the database of the product. . According to claim 15,
By executing the one or more instructions, the processor:
and registering the product as a new product in the database of the product when the first characteristic information and the second characteristic information do not match the database of the product. According to claim 11,
further comprising a communication interface;
By executing the one or more instructions, the processor:
An electronic device configured to receive, through the communication interface, the main encoder model and the product database from at least one server that learns the main encoder model and manages the product database. According to claim 11,
Including more cameras,
By executing the one or more instructions, the processor:
According to a user's input, an electronic device that executes a product recognition application and acquires an image related to the product through the camera. A computer-readable recording medium recording a program for executing the method of any one of claims 1 to 10 in a computer.

Images