KR102476498B1 - Method for identify product through artificial intelligence-based complex recognition and computer program recorded on record-medium for executing method therefor - Google Patents

Abstract

Proposed in the present invention is a product identification method through artificial intelligence-based complex recognition. According to the present invention, the recognition device may comprise the steps of: photographing at least one product through a camera; recognizing a barcode in the photographed image and shape information of a product included in the photographed image based on artificial intelligence (AI); and identifying the product included in the image based on the barcode and the shape information. Therefore, the accuracy of product identification can be improved.

Description

Method for identify product through artificial intelligence-based complex recognition and computer program recorded on record-medium for executing method therefor}

The present invention relates to product identification. More specifically, a product identification method for increasing the accuracy of product identification by complexly recognizing a barcode and product shape based on a vision sensor, and a computer recorded on a recording medium to execute the method. It's about the program.

Artificial intelligence (AI) refers to a technology that artificially implements some or all of human learning abilities, reasoning abilities, and perception abilities using computer programs. In relation to artificial intelligence (AI), machine learning refers to learning to optimize parameters with given data using a model composed of multiple parameters.

On the other hand, an unmanned store refers to a store that does not exist in a seller's store that performs payment and management of products, and allows consumers to bring goods they want to purchase and make direct payments.

Recently, interest in unmanned stores is increasing due to the development of IT (Information Technology) industrial technology, rising labor costs, and efficient operation in late-night hours. Such unmanned stores are mainly applied to stores that handle products that do not need to reflect various needs of customers, such as coffee shops, ice cream shops, etc., but are recently expanding their application area.

In such an unmanned store, an unmanned self-payment machine (for example, a kiosk) is installed so that customers can directly pay for the product they want to purchase, and various products and unmanned self-payment machines are installed in the unmanned store. Closed Circuit TeleVision (CCTV) is installed to monitor theft or damage.

In general, in an unmanned self-payment machine, a customer who wants to make a payment directly scans a barcode provided on the packaging of a product, identifies the product through the scanned barcode, and proceeds with payment using the information of the identified product. will do

Recently, when a customer who wants to make a payment places the item he/she wants to purchase on a specific location without directly scanning the barcode of the product, the barcode included in the product is automatically recognized based on the vision sensor to identify the product. Efforts are underway to do so.

However, since the barcode may harm the aesthetics of the product, it is printed to have a small, standardized size on the outer surface or wrapping paper of the product. Therefore, a barcode in an image captured by the vision sensor may be partially cut off, covered, distorted, or distorted, such as being out of focus. Accordingly, there is a problem in that the accuracy of product recognition is lowered.

In addition, in order to accurately recognize barcodes, a plurality of cameras for photographing products in various directions are required. However, since a lot of space is required to install a plurality of cameras, efforts for miniaturization are required.

Republic of Korea Patent Publication No. 10-2020-0023990, 'Payment processing system, method and computer program for unmanned stores', (published on March 6, 2020)

One object of the present invention is to provide a product identification method for increasing the accuracy of product identification by complexly recognizing a barcode and a shape of a product based on a vision sensor.

Another object of the present invention is to provide a computer program recorded on a recording medium to execute a product identification method for increasing the accuracy of product identification by complexly recognizing barcodes and product shapes based on a vision sensor.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to achieve the technical problem as described above, the present invention proposes a product identification method through artificial intelligence-based complex recognition. The method includes ~~~.

Specifically, the method includes the steps of a recognizing device photographing at least one product through a camera, wherein the recognizing device performs a barcode in the photographed image based on artificial intelligence (AI). ) and recognizing shape information of a product included in the photographed image, and identifying, by the recognizing device, a product included in the image based on the barcode and the shape information.

In the photographing step, the at least one product may be photographed through a plurality of cameras having different photographing positions.

The recognizing step recognizes the at least one product using the plurality of cameras, and performs product recognition based on an image captured by a camera having the highest product recognition rate among images captured by the plurality of cameras. can do.

In the photographing step, the camera may measure an image of the at least one product and a depth to the product included in the image.

In the identifying step, product groups of products included in the photographed image may be classified using a pre-learned classifier.

The identifying step measures the height value of the product included in the image through distance information to the product included in the image, compares the measured height value with an average height value of the classified product group, When the height value obtained is higher than the average height value of the product group, it may be determined that the products included in the image overlap.

In the identifying, the barcode may be searched within a product group identified based on the shape information.

In the identifying step, the artificial intelligence may include a barcode learning model learned based on barcodes and a shape learning model learned based on shape information.

In the identifying step, the recognized barcode is input into the barcode learning model, the recognized shape information is input into the shape learning model, and result values estimated from each of the barcode learning model and the shape learning model are combined. , the product can be identified.

In the photographing, a location where a barcode is expected to be located in the image may be estimated based on shape information of a product included in the captured image, and a distance value to the estimated location may be calculated.

In the photographing step, a region having shape information matching the shape information of a barcode defined in advance for a product may be estimated as a position where a barcode is expected to be located in the image by using a classifier learned in advance. .

The photographing step focuses the camera on the basis of the estimated position and the distance value to the estimated position and performs zooming to take the photograph, and the recognizing step re-recognizes the barcode in the captured image. can do.

In order to achieve the technical problem as described above, the present invention proposes a computer program recorded on a recording medium to execute a product identification method through artificial intelligence-based complex recognition. The computer program is combined with a computing device including a memory, a transceiver, and a processor that processes instructions resident in the memory, so that the processor operates at least one camera through a camera. Photographing a product, recognizing, by the processor, a barcode in the photographed image and shape information of the product included in the photographed image based on artificial intelligence (AI), and the processor A, a step of identifying a product included in the image based on the barcode and the shape information may be executed.

Details of other embodiments are included in the detailed description and drawings.

According to embodiments of the present invention, the accuracy of product identification may be increased by complexly recognizing a barcode of a product and a shape of the product in an image captured by a camera.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

1 is a schematic diagram showing the configuration of a product identification device according to an embodiment of the present invention.
2 is a schematic diagram showing the configuration of a product identification device according to another embodiment of the present invention.
3 is a logical configuration diagram of a product identification device according to an embodiment of the present invention.
4 is a logical configuration diagram of a product identification device according to another embodiment of the present invention.
5 is a logical configuration diagram of a product identification device according to another embodiment of the present invention.
6 is a logical configuration diagram of a product identification device according to another embodiment of the present invention.
7 is a hardware configuration diagram of an unattended payment device according to various embodiments of the present disclosure.
8 is a flowchart illustrating a product identification method according to an embodiment of the present invention.
9 is a flowchart illustrating a product identification method according to another embodiment of the present invention.
10 is a flowchart illustrating a product identification method according to another embodiment of the present invention.
11 is a flowchart illustrating a product identification method according to another embodiment of the present invention.
12 is an exemplary diagram for explaining a product identification method according to an embodiment of the present invention.
13 is an exemplary diagram for explaining a product identification method according to another embodiment of the present invention.
14 is an exemplary diagram for explaining a product identification method according to another embodiment of the present invention.

It should be noted that the technical terms used in this specification are only used to describe specific embodiments and are not intended to limit the present invention. In addition, technical terms used in this specification should be interpreted in terms commonly understood by those of ordinary skill in the art to which the present invention belongs, unless specifically defined otherwise in this specification, and are excessively inclusive. It should not be interpreted in a positive sense or in an excessively reduced sense. In addition, when the technical terms used in this specification are incorrect technical terms that do not accurately express the spirit of the present invention, they should be replaced with technical terms that those skilled in the art can correctly understand. In addition, general terms used in the present invention should be interpreted as defined in advance or according to context, and should not be interpreted in an excessively reduced sense.

Also, singular expressions used in this specification include plural expressions unless the context clearly indicates otherwise. In this application, terms such as 'consisting of' or 'having' should not be construed as necessarily including all of the various components or steps described in the specification, and some of the components or steps are included. It should be construed that it may not be, or may further include additional components or steps.

Also, terms including ordinal numbers such as first and second used in this specification may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention.

When a component is referred to as being 'connected' or 'connected' to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. On the other hand, when a component is referred to as "'directly connected' or 'directly connected' to another component, it should be understood that no other component exists in the middle.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings, but the same or similar components are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. In addition, in describing the present invention, if it is determined that a detailed description of a related known technology may obscure the gist of the present invention, the detailed description will be omitted. In addition, it should be noted that the accompanying drawings are only for easily understanding the spirit of the present invention, and should not be construed as limiting the spirit of the present invention by the accompanying drawings. The spirit of the present invention should be construed as extending to all changes, equivalents or substitutes other than the accompanying drawings.

On the other hand, recently, when a customer who wants to make a payment places an item he wants to purchase on a specific location without directly scanning the barcode of the product, the barcode included in the product is automatically recognized based on the vision sensor. Efforts are underway to identify

However, since the barcode may harm the aesthetics of the product, it is printed to have a small, standardized size on the outer surface or wrapping paper of the product. Therefore, a barcode in an image captured by the vision sensor may be partially cut off, covered, distorted, or distorted, such as being out of focus. Accordingly, there is a problem in that the accuracy of product recognition is lowered.

In addition, in order to accurately recognize barcodes, a plurality of cameras for taking pictures of products in various directions are essential. However, since a lot of space is required to install a plurality of cameras, efforts for miniaturization are required.

In order to overcome these limitations, the present invention is capable of increasing the accuracy of product identification and proposes various means for miniaturizing a device for identifying products.

Hereinafter, a product identification device according to an embodiment of the present invention will be described.

1 is a schematic diagram showing the configuration of a product identification device according to an embodiment of the present invention.

Referring to FIG. 1 , a product identification device 100 according to an embodiment of the present invention may be installed in an unmanned store, identify a product, and may be a device for performing payment for the identified product. The unmanned store to which the product identification device 100 is applied includes a freezer/refrigerator for fresh product storage, a dessert display space, a vegan and organic product display space, a refrigerated food display space, a new product display space, a room temperature storage product display space, and a product identification device. (100) can be configured as a space to be displayed. Here, the products stored in the unmanned store may be products to which barcodes are attached, products to which barcodes are not attached, standard products, or non-standard products.

The product identification device 100 identifies the product selected by the purchaser, extracts payment information related to the product, and extracts payment information related to the product when the purchaser enters the unmanned store, selects a desired product, and places the selected product in a specific space. Payment can be made through payment information.

The product identification device 100 may photograph at least one product through a camera and crop a barcode area within the photographed image. Here, cropping means an operation of removing an area other than a barcode area. Thereafter, the product identification device 100 may identify at least one feature point in the cropped barcode area. The product identification device 100 may restore a barcode area using feature points identified based on artificial intelligence (AI), and recognize information from the restored barcode area to identify a product.

In addition, the product identification device 100 photographs at least one product through a camera, and displays a barcode in the captured image based on artificial intelligence (AI) and the shape of the product included in the captured image. information can be recognized. The product identification device 100 may identify a product included in an image based on the recognized barcode and shape information.

Also, the product identification device 100 may move at least one product including a plurality of barcodes printed with a paint having a higher reflectance than a preset value for light in a specific wavelength range through a plurality of conveyors. The product identification device 100 irradiates light in a specific wavelength range to at least one moving product through a sensing unit disposed in a slit formed between a plurality of conveyors to sense at least one of barcodes printed on the product. have. Also, the product identification device 100 may identify the product by recognizing the sensed barcode. A barcode invisible to the human eye can be printed by using a paint having a higher reflectance than a preset value for light in a specific wavelength range excluding the visible ray region. Existing barcodes were provided in a limited size in a limited location in order not to impair the aesthetics of product packaging, but by using invisible paint, it can be used anywhere on product packaging without size restrictions . Since multiple barcodes of various shapes can be provided at any location on the product packaging without limiting the number, the act of bringing the barcode-located part close to the sensing unit is omitted in order to recognize the barcode, and a fixed sensing unit is used to detect the product. At least one barcode among a plurality of barcodes present at an unspecified location can be more easily recognized.

In addition, the product identification device 100 photographs at least one product through a plurality of cameras that detect light in different wavelength ranges, and captures a plurality of images taken from the plurality of cameras based on artificial intelligence (AI). At least one product may be identified based on shape information of at least one product included in the image.

The product identification device 100 may perform a payment such as cash payment or credit card payment for the identified product. Here, when performing card payment, the product identification device 100 may perform payment by transmitting and receiving data using a network combining at least one of a card company server and a security line, a public wired communication network, or a mobile communication network through a communication network. there is.

For example, public wired communication networks may include Ethernet, x Digital Subscriber Line (xDSL), Hybrid Fiber Coax (HFC), and Fiber To The Home (FTTH). It may be, but is not limited thereto. In addition, in the mobile communication network, Code Division Multiple Access (CDMA), Wideband CDMA (WCDMA), High Speed Packet Access (HSPA), Long Term Evolution, LTE) and 5th generation mobile telecommunication may be included, but is not limited thereto.

Describing each component, the product identification device 100 according to an embodiment of the present invention may include a main body 110, a transfer unit 120, and a photographing unit 130.

The main body 110 constitutes the body of the product identification device 100, and may form a space for installing the transfer unit 120 and the photographing unit 130. Here, the main body 110 is shown as being formed in a hexahedral shape, but therein, a space in which hardware for driving the product identification device 100 is inserted and a space for installing the transfer unit 120 and the photographing unit 130 is formed. It can be formed in various shapes that can be made.

The transfer unit 120 may move the product in the direction of the photographing unit 130 so that the product can be recognized through the photographing unit 130 when a buyer places a product to be purchased.

The transfer unit 120 may include a conveyor to move products. Here, various conveyors capable of moving products such as a belt conveyor, a screw conveyor, a bucket conveyor, a roller conveyor, and a trolley conveyor may be applied. .

The photographing unit 130 may generate an image by photographing the product in order to recognize at least one product moving through the transfer unit 120 . Such a photographing unit 130 may be located on a path along which products are moved by the conveying unit 120 .

Here, the photographing unit 130 may include at least one vision sensor. The vision sensor may be at least one of a visible ray camera, an infrared camera, a thermal imaging camera, a depth camera, a lidar sensor, a radar sensor, and an ultrasonic sensor.

Hereinafter, a product identification device 200 according to another embodiment of the present invention will be described.

2 is a schematic diagram showing the configuration of a product identification device according to another embodiment of the present invention.

Referring to FIG. 2 , a product identification device 200 according to another embodiment of the present invention may include a transfer unit 220 and a sensing unit 230 .

Referring to each component, the transfer unit 220 may move at least one product on which a barcode is printed. The transfer unit 220 may include a plurality of conveyors 221 and 222 . For example, the transfer unit 220 may include a first conveyor 221 and a second conveyor 222 . Here, a slit forming a space for sensing by the sensing unit 230 may be formed between the first conveyor 221 and the second conveyor 222 . That is, the first conveyor 221 and the second conveyor 222 are spaced apart at regular intervals to form a space through which light emitted from the sensing unit 230 passes.

In the following configuration, the sensing unit 230 is disposed in a slit formed between the first conveyor 221 and the second conveyor 222, and at least one moving by the first conveyor 221 and the second conveyor 222. At least one of a plurality of barcodes printed on the product may be sensed by irradiating the product with light having a specific wavelength range.

Here, at least one product may include a plurality of barcodes printed with paint having a reflectance higher than a preset value for light in a specific wavelength range.

For example, a plurality of barcodes may be printed with a paint having higher reflectivity than a preset value for light in a specific wavelength range other than 400 to 700 nm, but printed at preset intervals on the front surface of a product.

On the other hand, since the barcode may harm the aesthetics of the product, it is printed to have a small, standardized size on the outer surface or wrapping paper of the product.

Accordingly, since the barcode according to an embodiment of the present invention is printed with a paint having a higher reflectance than a preset value for a specific wavelength range excluding the visible ray region of 400 to 700 nm visible to the human eye, it is freely displayed on the front of the product. can be printed

For example, a barcode may be printed using a paint containing a perovskite-type complex oxide. Here, the paint is at least an alkaline earth metal element, at least one element selected from titanium, zirconium, and niobium, and, if necessary, manganese (Mg) or iron (Fe); A perovskite-type composite oxide including an element of group IIIa of the periodic table such as aluminum (Al) and gallium (Ga) or zinc (Zn) may be included. These paints have sufficient reflectance for infrared rays in the wavelength range of 700 to 1050 nm, and have excellent thermal stability and heat resistance, thereby reducing concerns about safety and environmental issues.

Barcodes of different shapes may be printed according to product categories. For example, barcode information may be printed in a “+”-shaped area in the case of fresh products, and barcode information may be printed in an “o”-shaped area in the case of industrial products.

Also, the barcodes may be printed at intervals set in advance according to product categories. For example, barcodes may be printed at intervals of 5 cm for fresh products and at intervals of 3 cm for industrial products.

In addition, barcodes may be printed in preset patterns according to product categories. For example, the barcode may be printed by arranging barcodes having different sizes in a predetermined pattern.

The barcode may be directly printed on the outer surface of the product or wrapping paper, or may be made in the form of a sheet or film and attached to the product.

The sensing unit 230 for sensing the barcode includes a light irradiation unit 231 that irradiates light in a specific wavelength range and a camera 232 that photographs at least one product when light is irradiated from the light irradiation unit 231. can be configured to include That is, in a state in which light is irradiated by the light irradiation unit 231, the camera 232 may perform photographing.

When a barcode is printed using the above-described perovskite-type complex oxide, the light irradiator 231 irradiates light in a wavelength range of 700 to 1050 nm so that the barcode is recognized by the camera 232 . However, it is not limited thereto, and light of various wavelength ranges may be irradiated to match the characteristics of the paint for barcode printing. Meanwhile, the light irradiation unit 231 and the camera 232 are illustrated as being separated from each other, but may be implemented as a single device.

The camera 232 may be a camera capable of capturing a range of wavelengths of light irradiated by the light irradiation unit 231 . For example, when the light irradiation unit 231 emits light in a wavelength range of 700 to 1050 nm, an infrared camera may be used as the camera 232 .

Hereinafter, a logical configuration of a product identification device according to various embodiments of the present invention will be described in detail.

3 is a logical configuration diagram of a product identification device according to an embodiment of the present invention.

As shown in FIG. 3, the product identification device according to an embodiment of the present invention includes a communication unit 305, an input/output unit 310, a storage unit 315, a cropping unit 320, a feature point identification unit 325, It may be configured to include a barcode restoration unit 330 and a product identification unit 335.

Since the components of the product identification device are merely functionally distinct elements, two or more components may be integrated and implemented in an actual physical environment, or one component may be implemented separately from each other in an actual physical environment. You will be able to.

Each component will be described. The communication unit 305 can transmit/receive data to an external card company server, a point providing server, etc. when payment is made using a card, points, etc. for payment of a recognized product. .

Specifically, the communication unit 305 may transmit payment information for a recognized product for which payment is to be made to the outside, and may receive a payment processing result according to product payment.

With the following configuration, the input/output unit 310 may receive a signal from a user through a user interface (UI) or output an operation result to the outside.

Specifically, the input/output unit 310 outputs an identification result for a product to be purchased through a screen, outputs payment information for the identified product through a screen, and displays payment processing information after payment is completed through a screen. can be printed out.

The storage unit 315 may store data required for operation of the product identification device.

Specifically, the storage unit 315 may include a buffer for storing a program for identifying a product and a program for performing product payment.

With the following configuration, the cropping unit 320 may crop a barcode area of a product within an image captured by a camera. Here, cropping means an operation of removing an area other than a barcode area.

The cropping unit 320 may identify at least one product by removing a background of the photographed image and partitioning the foreground according to an outline. That is, in the case of a plurality of products, the cropping unit 320 removes the remaining background except for the area related to the product in the captured image, and separates each product by dividing the area related to the product according to the outline.

The cropping unit 320 may crop the barcode area for each of the at least one classified product. That is, the cropping unit 320 has a shape that matches the shape information of a barcode defined in advance for each product classified using a classifier learned in advance based on artificial intelligence (AI). The area can be recognized as a barcode area.

With the following configuration, the feature point identification unit 325 can identify at least one feature point in the barcode area. That is, the feature point identification unit 325 may identify barcode information included in the barcode area, and identify a damage type of the barcode according to the number and arrangement relationship of the identified barcode information.

Here, the barcode information may include at least one of quiet zone, start/stop character, check digit, interpretation line, bar/space, and intercharacter gaps included in the barcode area. However, it is not limited thereto, and barcode information may be set in various ways according to predetermined rules.

The damage type may include at least one of obscuring, cutting, twisting, and out of focus of the barcode area. However, it is not limited thereto, and the damage type may correspond to various distortion types that may occur in an image captured by a vision sensor.

The barcode restoration unit 330 may restore a barcode area using feature points identified based on artificial intelligence (AI).

That is, the barcode restoration unit 330 compares the similarity between the feature points of the pre-stored barcodes and the feature points identified from the barcode area, and restores the barcode area based on a barcode whose similarity is equal to or greater than a preset threshold among the pre-stored barcodes. . For example, the barcode restoration unit 330 extracts identifiable barcode information among quiet zone, start/stop character, check digit, interpretation line, bar/space, and intercharacter gaps in the barcode, and has the most similarity with the extracted barcode information. A barcode with a high value may be extracted from the storage unit 315, and a damaged or distorted portion of the barcode area may be restored based on the extracted barcode.

In addition, the barcode restoration unit 330 divides the barcode area into an upper area and a lower area, and expands bar/space to the upper or lower area based on the upper or lower area to restore the barcode area. . That is, the bar/space of the barcode has a linear structure from the lower area to the upper area or from the upper area to the lower area due to its characteristics. Accordingly, the barcode restoration unit 330 may restore the barcode area by dividing the upper area and the lower area in the barcode area and complementing the upper area and the lower area.

In addition, the barcode restoration unit 330 may restore the barcode area by inputting the identified barcode information and damage type to machine learning artificial intelligence (AI) in advance according to the barcode information and damage type. there is.

In addition, the barcode restoration unit 330 may restore the damaged area based on pixels of undamaged areas within the barcode area based on a generative adversarial network (GAN).

Here, the generative adversarial network is a structure in which a generator and a discriminator are trained competitively to a level where it is difficult to distinguish whether data created by the generator is real or fake.

Here, the discriminator learns using real learning data and fake data created by the generator, and plays a role in distinguishing whether the input data is real or fake, and the generator plays a role in creating fake data using the probability density of real data.

The barcode restoration unit 330 may reset the weight by correcting the error while transmitting the output result value as an input value by using error back-propagation to the generative adversarial neural network. Here, the barcode restoration unit 330 may restore the barcode area by assigning a loss weight for restoration to pixels in the undamaged area based on the distance from the damaged area. At this time, the barcode restoration unit 330 may assign a weight for loss for restoration in proportion to the number of damaged pixels existing around each pixel in the undamaged area. That is, it is possible to increase the accuracy of restoration by allowing pixels far from the damaged portion to have a small effect and pixels close to the damaged portion to have a high effect.

With the following configuration, the product identification unit 335 may identify the product by recognizing information from the restored barcode area. That is, when the product identification unit 335 can recognize desired barcode information from the restored barcode area, it can identify the product using barcode information acquired from only the restored barcode area. In addition, when the product identification unit 335 does not have enough information to recognize the barcode information from the restored barcode area, the product identification unit 335 extracts a barcode whose similarity with the restored barcode area is equal to or greater than a preset value, and identifies the product from the extracted barcode. can do.

4 is a logical configuration diagram of a product identification device according to another embodiment of the present invention.

As shown in FIG. 4 , the product identification device according to another embodiment of the present invention includes a communication unit 405, an input/output unit 410, a storage unit 415, a barcode and shape information recognition unit 420, and a product identification unit. It may be configured to include (430).

Since the components of the product identification device are merely functionally distinct elements, two or more components may be integrated and implemented in an actual physical environment, or one component may be implemented separately from each other in an actual physical environment. You will be able to.

Each of the components will be described. The communication unit 405 can transmit/receive data to an external card company server, a point providing server, etc. when payment is made using a card, points, etc. for payment of a recognized product. .

Specifically, the communication unit 405 may transmit payment information for a recognized product to be paid to the outside and receive a payment processing result according to product payment.

With the following configuration, the input/output unit 410 may receive a signal from a user through a user interface (UI) or output an operation result to the outside.

Specifically, the input/output unit 410 outputs an identification result for a product to be purchased through a screen, outputs payment information for the identified product through a screen, and displays payment processing information after payment is completed through a screen. can be printed out.

The storage unit 415 may store data required for operation of the product identification device.

Specifically, the storage unit 415 may include a buffer for storing a program for identifying a product and a program for performing product payment.

With the following configuration, the barcode and shape information recognizing unit 420 photographs at least one product through a camera, and captures a barcode and image within the photographed image based on artificial intelligence (AI). It is possible to recognize the shape information of the product included in the image.

Here, the barcode and shape information recognizing unit 420 may photograph at least one product through a plurality of cameras having different photographing positions. The barcode and shape information recognition unit 420 recognizes at least one product using a plurality of cameras, but recognizes the product based on an image captured by a camera having the highest product recognition rate among images captured by the plurality of cameras. can be performed. That is, the barcode and shape information recognizing unit 420 may calculate a recognition rate for the shape and barcode of each product captured by a plurality of cameras, and perform product recognition based on the image having the highest recognition rate. However, it is not limited thereto, and the barcode and shape information recognizing unit 420 may generate one image by mutually complementing images taken from a plurality of cameras, and perform product recognition using the generated image.

Also, the barcode and shape information recognizing unit 420 may measure an image of at least one product and a depth to a product included in the image. Through this, the barcode and shape information recognizing unit 420 may assist in determining whether products overlap.

In addition, the barcode and shape information recognition unit 420 estimates a position where the barcode is expected to be located in the image based on the shape information of the product included in the captured image, and calculates a distance value to the estimated position. can Here, the barcode and shape information recognition unit 420 uses a pre-learned classifier to designate an area having shape information that matches the formation information of a barcode defined in advance for a product in an image where a barcode is expected to be located. location can be estimated.

Also, the barcode and shape information recognizing unit 420 may focus the camera based on the estimated position and the distance value to the estimated position, and perform zooming to take a picture. That is, since the barcode and shape information recognizing unit 420 is printed in a small size due to the nature of the barcode, when the overall shape of the product is focused and photographed, the focus of the barcode is out of focus, and the barcode recognition rate may be significantly lowered. Accordingly, the barcode and shape information recognizing unit 420 focuses on the estimated barcode position and re-recognizes the barcode through the recaptured image, thereby increasing the barcode recognition rate.

With the following configuration, the product identification unit 425 may identify the product included in the image based on the barcode and shape information.

Meanwhile, when recognizing a product through a vision sensor, since the product is recognized based on an image captured at at least one fixed point, it is necessary to consider overlapping products.

Accordingly, the product identification unit 425 may classify product groups of products included in the photographed image using a pre-learned classifier. Here, the product identification unit 425 measures the height value of the product included in the image through distance information to the product included in the image, compares the measured height value with the average height value of the classified product groups, If the height value is higher than the average height value of the product group, it may be determined that the products included in the image overlap. Then, when it is determined that the products overlap, the product identification unit 425 may output whether or not the products overlap in the form of sound through the input/output unit 410 . However, it is not limited thereto, and the product identification unit 425 may output text for confirming whether or not overlapping or output an image focused on overlapping products.

Also, the product identification unit 425 may identify a product group based on the shape information and search for a barcode within the identified product group. For example, the product identification unit 425 can significantly increase the speed and accuracy of product identification by identifying a product group through the shape of the product included in the image and performing a barcode search within the identified product group.

In addition, the product identification unit 425 may perform product identification through a barcode learning model learned based on barcodes and a shape learning model learned based on shape information.

That is, the product identification unit 425 inputs the recognized barcode to the barcode learning model, inputs the recognized shape information to the shape learning model, and synthesizes the result values estimated from each of the barcode learning model and the shape learning model, product can be identified.

For example, the product identification unit 425 may identify a product through a result value estimated from a learning model having a higher recognition rate than a preset value among a barcode learning model and a shape learning model.

In addition, the product identification unit 425 may input images to the barcode learning model and the shape learning model a preset number of times, and identify the most aggregated result value among the estimated result values as a product.

5 is a logical configuration diagram of a product identification device according to another embodiment of the present invention.

As shown in FIG. 5 , the product identification device according to another embodiment of the present invention includes a communication unit 505, an input/output unit 510, a storage unit 515, a category classification unit 520, and a product identification unit 525. It can be configured to include.

Since the components of the product identification device are merely functionally distinct elements, two or more components may be integrated and implemented in an actual physical environment, or one component may be implemented separately from each other in an actual physical environment. You will be able to.

Each of the components will be described. The communication unit 505 can transmit/receive data to an external card company server, a point providing server, etc. when payment is made using a card, points, etc. for a recognized product. .

Specifically, the communication unit 505 may transmit payment information for a recognized product to be paid to the outside, and may receive a payment processing result according to product payment.

With the following configuration, the input/output unit 510 may receive a signal from a user through a user interface (UI) or output an operation result to the outside.

Specifically, the input/output unit 510 outputs an identification result for a product to be purchased through a screen, outputs payment information for the identified product through a screen, and displays payment processing information after payment is completed through a screen. can be printed out.

The storage unit 515 may store data required for operation of the product identification device.

Specifically, the storage unit 515 may include a buffer for storing a program for classifying product categories, a program for identifying products, and a program for performing product payment.

With the following configuration, the category classification unit 520 classifies product categories based on the sensed barcode shape of at least one product, recognizes barcode information included in the barcode, and barcode information matched with the barcode information within the category. By searching for, you can identify the product.

Here, at least one product may include a plurality of barcodes printed with paint having higher reflectance than a preset value for light in a specific wavelength range.

The plurality of barcodes may be printed with a paint having higher reflectance than a preset value for light in a specific wavelength range except for 400 to 700 nm, and may be printed on the front surface of the product at preset intervals.

That is, since the barcode may harm the aesthetics of the product, it is printed to have a small, standardized size on the outer surface or wrapping paper of the product.

Accordingly, since the barcode according to an embodiment of the present invention is printed with a paint having a higher reflectance than a preset value for a specific wavelength range excluding the visible ray region of 400 to 700 nm visible to the human eye, it is freely displayed on the front of the product. It can be printed, and the size can also be printed freely.

The category classification unit 520 may estimate the size of a barcode based on the number of barcodes recognized for a preset time from the time the product is recognized, and classify the product category according to the estimated size.

Also, the category classification unit 520 may classify product categories based on intervals between a plurality of barcodes. In this case, the barcodes may be spaced apart at predetermined intervals according to product categories and printed. For example, barcodes may be printed at intervals of 5 cm for fresh products and at intervals of 3 cm for industrial products.

Also, the category classification unit 520 may classify product categories based on a plurality of barcode print patterns. In this case, the barcode may be printed in a preset pattern according to the product category. For example, the barcode may be printed by arranging barcodes having different sizes in a predetermined pattern.

With the following configuration, the product identification unit 525 may search for barcode information that matches the barcode information within the category classified by the category classification unit 520 to identify the product.

6 is a logical configuration diagram of a product identification device according to another embodiment of the present invention.

As shown in FIG. 6 , a product identification device according to another embodiment of the present invention may include a communication unit 605, an input/output unit 610, a storage unit 615, and a product identification unit 620. there is.

Since the components of the product identification device are merely functionally distinct elements, two or more components may be integrated and implemented in an actual physical environment, or one component may be implemented separately from each other in an actual physical environment. You will be able to.

Each component will be described. The communication unit 605 can transmit/receive data to an external card company server, a point providing server, etc. when a payment is made using a card, points, etc. for payment of a recognized product. .

Specifically, the communication unit 605 may transmit payment information for a recognized product to be paid to the outside and receive a payment processing result according to product payment.

With the following configuration, the input/output unit 610 may receive a signal from a user through a user interface (UI) or output an operation result to the outside.

Specifically, the input/output unit 610 outputs an identification result for a product to be purchased through a screen, outputs payment information for the identified product through a screen, and displays payment processing information after payment is completed through a screen. can be printed out.

The storage unit 615 may store data required for operation of the product identification device.

Specifically, the storage unit 615 may include a buffer for storing a program for identifying a product and a program for performing product payment.

With the following configuration, the product identification unit 620 identifies at least one product based on artificial intelligence (AI) based on shape information of at least one product included in a plurality of images taken from a plurality of cameras. can be identified.

Here, the plurality of cameras may be two or more cameras selected from among a visible ray camera, an infrared camera, and a thermal imaging camera. However, it is not limited thereto, and the plurality of cameras may include at least one of a depth camera, a lidar, a radar, and an ultrasonic sensor.

The product identification unit 620 may restore an image captured by a visible ray camera to an image captured by an infrared camera or a thermal imaging camera. However, it is not limited thereto, and the product identification unit 620 may generate an image with a high recognition rate through various combinations, such as restoring an image captured by an infrared camera to an image captured by a visible ray camera or a thermal imaging camera. .

In addition, the product identification unit 620 includes a first learning model learned based on the shape information of the product photographed by the visible ray camera, and a first learned model based on the shape information of the product photographed by the infrared camera. The product may be identified based on the second learning model and a third learning model learned based on shape information of the product photographed by the thermal imaging camera.

Here, the product identification unit 620 inputs an image taken from a camera corresponding to each learning model to two or more learning models selected from among the first learning model, the second learning model, and the third learning model, and calculates the estimated result value. Taken together, the product can be identified.

In addition, the product identification unit 620 identifies the product by synthesizing the result values estimated from the learning model having a higher recognition rate than the preset value among the result values estimated from the first learning model, the second learning model, and the third learning model. can do.

In addition, the product identification unit 620 inputs an image of a camera corresponding to each learning model to two or more learning models selected from among the first learning model, the second learning model, and the third learning model a preset number of times, and the estimated result Among the values, the most aggregated result value can be identified as a product.

In addition, the product identification unit 620 may recognize a barcode in the captured image based on the image captured by the visible ray camera. Here, the product identification unit 620 may classify a product category based on an image captured by a thermal imaging camera, search for barcode information recognized within the category, and identify the product.

In addition, the product identification unit 620 may overlay a plurality of images taken from a plurality of cameras at the same location and identify the product based on the overlaid image. Then, the product identification unit 620 removes the background of the overlaid image, divides the foreground according to the outline, classifies at least one product, and classifies each product into one of irregular products and standard products. can be distinguished. That is, the product identification unit 620 may support more clear distinction of outlines by overlaying a plurality of images taken at the same location from different types of cameras.

In addition, the product identification unit 620 may identify the product based on shape information of the product when the product is classified as an irregular product, and may identify the product based on the barcode of the product when classified as a standard product. That is, in the case of an irregular product, even if the barcode is printed on the packaging material, since the shape is irregular, the barcode recognition rate is remarkably low. so high Accordingly, the product identification unit 620 distinguishes irregular products from standard products, identifies products based on shape information in the case of irregular products having a relatively low barcode recognition rate, and barcodes in the case of standard products having a relatively high barcode recognition rate. Products can be identified through information.

7 is a hardware configuration diagram of an unattended payment device according to various embodiments of the present disclosure.

As shown in FIG. 7, the product identification device includes a processor (Processor, 150), a memory (Memory, 155), a transceiver (160), an input/output device (165), and a data bus (Bus, 170). ) and storage (Storage, 175).

The processor 150 may implement operations and functions of the product identification device based on instructions according to the software 180a in which the method according to the embodiments of the present invention is resident in the memory 155 . Software 180a in which a method according to embodiments of the present invention is implemented may be loaded in the memory 155 .

The transceiver 160 may transmit/receive data to an external card company server or a point providing server when payment is performed using a card, points, or the like, for payment of recognized products. That is, the transceiver 160 may transmit payment information for a recognized product to be paid for and receive a payment processing result according to product payment.

The data bus 170 is connected to the processor 150, the memory 155, the transceiver 160, the input/output device 165, and the storage 175, and is a movement path for transferring data between each component. role can be fulfilled.

The storage 175 stores an application programming interface (API), a library file, a resource file, etc. necessary for the execution of the software 180a in which the method according to the embodiments of the present invention is implemented. can be saved The storage 175 may store the software 180b and the database 185 in which a method according to embodiments of the present invention is implemented.

According to one embodiment of the present invention, the software (180a, 180b) for implementing the control method of the sensors resident in the memory 155 or stored in the storage 175, the processor 150 via the camera (camera) at least Photographing one product, cropping the barcode area of the at least one product within the captured image, identifying at least one feature point in the barcode area, In order to execute the step of restoring the barcode area using the feature points identified based on artificial intelligence (AI) and the step of recognizing information from the restored barcode area and identifying the product, a computer recorded on a recording medium can be a program.

In addition, according to another embodiment of the present invention, the step of photographing at least one product through a camera, the barcode and the captured image within the photographed image based on artificial intelligence (AI) It may be a computer program recorded on a recording medium to execute the step of recognizing the shape information of the product included in the image and the step of identifying the product included in the image based on the barcode and shape information.

Further, according to another embodiment of the present invention, at least one conveyor including a plurality of conveyors includes a plurality of barcodes printed with a paint having a higher reflectance than a preset value for light in a specific wavelength range. Moving the product of the step, a sensing unit disposed in a slit formed between a plurality of conveyors irradiates at least one product moving by the transfer unit with light in a specific wavelength range, and at least one of a plurality of barcodes printed on the product In order to execute the step of sensing one and the step of identifying the product by recognizing the sensed barcode by the controller, it may be a computer program recorded on a recording medium.

In addition, according to another embodiment of the present invention, photographing at least one product through a plurality of cameras detecting light of different wavelength ranges and from a plurality of cameras based on artificial intelligence (Artificial Intelligence, AI) It may be a computer program recorded on a recording medium to execute a step of analyzing a plurality of captured images and identifying at least one product through shape information on at least one product included in the plurality of images.

More specifically, the processor 150 may include an Application-Specific Integrated Circuit (ASIC), another chipset, a logic circuit, and/or a data processing device. The memory 155 may include read-only memory (ROM), random access memory (RAM), flash memory, memory cards, storage media, and/or other storage devices. The transceiver 160 may include a baseband circuit for processing wired/wireless signals. The input/output device 165 includes an input device such as a keyboard, a mouse, and/or a joystick, and a Liquid Crystal Display (LCD), an Organic LED (OLED), and/or a liquid crystal display (LCD). Alternatively, an image output device such as an active matrix OLED (AMOLED) may include a printing device such as a printer or a plotter.

When the embodiments included in this specification are implemented as software, the above-described method may be implemented as a module (process, function, etc.) that performs the above-described functions. A module may reside in memory 255 and be executed by processor 150 . The memory 155 may be internal or external to the processor 150 and may be connected to the processor 150 by various well-known means.

Each component shown in FIG. 7 may be implemented by various means, eg, hardware, firmware, software, or a combination thereof. In the case of hardware implementation, one embodiment of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), FPGAs ( Field Programmable Gate Arrays), processors, controllers, microcontrollers, microprocessors, etc.

In addition, in the case of implementation by firmware or software, an embodiment of the present invention is implemented in the form of a module, procedure, function, etc. that performs the functions or operations described above, and is stored on a recording medium readable through various computer means. can be recorded. Here, the recording medium may include program commands, data files, data structures, etc. alone or in combination. Program instructions recorded on the recording medium may be those specially designed and configured for the present invention, or those known and usable to those skilled in computer software. For example, recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs (Compact Disk Read Only Memory) and DVDs (Digital Video Disks), floptical It includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, such as a floptical disk, and ROM, RAM, flash memory, and the like. Examples of program instructions may include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. These hardware devices may be configured to operate as one or more pieces of software to perform the operations of the present invention, and vice versa.

Hereinafter, a product identification method according to various embodiments of the present invention will be described.

8 is a flowchart illustrating a product identification method according to an embodiment of the present invention.

Referring to FIG. 8 , the product identification device may photograph at least one product moving by the conveyor (S110). Here, the product identification device may photograph the product through a plurality of cameras, and may determine an image having the highest barcode recognition rate among the plurality of cameras.

Next, the product identification device may crop a barcode area of the product within the image captured by the camera (S120).

That is, the product identification device may identify at least one product by removing a background of the photographed image and partitioning the foreground according to an outline. That is, in the case of a plurality of products, the product identification device may identify each product by removing the remaining background except for an area related to the product in the photographed image and dividing the area related to the product according to an outline.

Also, the product identification device may crop a barcode area for each of the at least one classified product. That is, the product identification device is an area having shape information that matches the shape information of a predefined barcode for each product classified using a classifier learned in advance based on artificial intelligence (AI). can be recognized as a barcode area.

Next, the product identification device may identify at least one feature point in the barcode area (S130). That is, the product identification device may identify barcode information included in the barcode area, and identify a damage type of the barcode according to the number and arrangement relationship of the identified barcode information.

Here, the barcode information may include at least one of quiet zone, start/stop character, check digit, interpretation line, bar/space, and intercharacter gaps included in the barcode area. However, it is not limited thereto, and barcode information may be set in various ways according to predetermined rules.

The damage type may include at least one of obscuring, cutting, twisting, and out of focus of the barcode area. However, it is not limited thereto, and the damage type may correspond to various distortion types that may occur in an image captured by a vision sensor.

Next, the product identification device may restore the barcode area using the feature points identified based on artificial intelligence (AI) (S140).

That is, the product identification device may compare similarities between feature points of pre-stored barcodes and feature points identified from the barcode area, and restore the barcode area based on a barcode whose similarity is equal to or greater than a preset threshold among pre-stored barcodes.

Also, the product identification device may divide the barcode area into an upper area and a lower area, and may restore the barcode area by extending bar/space to the upper or lower area based on the upper or lower area. That is, the bar/space of the barcode has a linear structure from the lower area to the upper area or from the upper area to the lower area due to its characteristics. Accordingly, the product identification device may restore the barcode area by dividing the upper area and the lower area in the barcode area and complementing the upper and lower areas.

In addition, the product identification device may restore the barcode area by inputting the identified barcode information and damage type to machine learning artificial intelligence (AI) in advance according to the barcode information and damage type.

Also, the product identification device may restore the damaged area based on pixels of undamaged areas within the barcode area based on a generative adversarial network (GAN).

The product identification device may reset the weight by correcting the error while transmitting the output result value as an input value by using error back-propagation to the generative adversarial neural network. Here, the product identification device may restore the barcode area by assigning a loss weight for restoration to pixels in the undamaged area based on the distance from the damaged area. At this time, the product identification device may assign a weight to the loss for restoration in proportion to the number of damaged pixels existing around each pixel of the non-damaged area. That is, it is possible to increase the accuracy of restoration by allowing pixels far from the damaged part to have a small effect and pixels close to the damaged part to have a high effect.

Next, the product identification device may identify the product by recognizing information from the restored barcode area (S150). That is, when the product identification device can recognize desired barcode information from the restored barcode area, it can identify the product using barcode information obtained from only the restored barcode area. In addition, when information for recognizing barcode information is not sufficient from the restored barcode area, the product identification device extracts a barcode whose similarity to the restored barcode area is equal to or greater than a preset value, and identifies the product from the extracted barcode. .

9 is a flowchart illustrating a product identification method according to another embodiment of the present invention.

Referring to FIG. 9 , the product identification device may photograph at least one product moving on the conveyor through a camera (S210). Here, the camera may photograph at least one product through a plurality of cameras having different photographing positions. Here, the product identification device may determine an image captured by a camera having the highest product recognition rate among images captured by a plurality of cameras.

Also, the product identification device may measure a depth to a product included in the image. Here, the camera may be a depth camera capable of measuring a distance to a product included in the image.

Next, the product identification device determines whether the product's barcode recognition rate is greater than or equal to a preset value (S220). That is, the product identification device may calculate barcode recognition rates of images photographed by a plurality of cameras and compare them with a preset value.

Here, when the recognition rate of the plurality of images is lower than a preset value, the product identification device estimates a position where the barcode is expected to be located in the image based on shape information of the product included in the photographed image, and The distance value to the position can be calculated. At this time, the product identification device may estimate a region having shape information matching predefined barcode formation information for a product as a position where a barcode is expected to be located in the image by using a pre-learned classifier. .

The product identification device may focus the camera based on the estimated position and the distance value to the estimated position, and perform zooming to take a picture (S230). That is, since barcodes are printed in a small size in the product identification device, when the overall shape of a product is focused and photographed, the focus of the barcode is out of focus, and the barcode recognition rate may be remarkably lowered. Accordingly, the product identification device can increase the barcode recognition rate by focusing on the estimated barcode position and re-recognizing the barcode through the recaptured image.

Next, the product identification device may recognize a barcode in the captured image based on artificial intelligence (AI) and shape information of the product included in the captured image (S240).

Then, the product identification device may identify the product included in the image based on the barcode and shape information (S250).

Meanwhile, when recognizing a product through a vision sensor, since the product is recognized based on an image captured at at least one fixed point, it is necessary to consider overlapping products.

Accordingly, the product identification device may classify product groups of products included in the photographed image using a pre-learned classifier. Here, the product identification device measures the height value of the product included in the image through information on the distance to the product included in the image, compares the measured height value with the average height value of the classified product group, and obtains the measured height value. If it is higher than the average height value of the product group, it may be determined that the products included in the image overlap. Thereafter, when it is determined that the products overlap, the product identification device may output whether or not the products overlap in the form of a sound. However, the product identification device is not limited thereto, and the product identification device may output text for confirming whether or not the product is overlapped or output an image focused on the overlapped products.

Also, the product identification device may identify a product group based on the shape information and search for a barcode within the identified product group. For example, the product identification device can significantly increase the speed and accuracy of product identification by identifying a product group through the shape of the product included in the image and performing a barcode search within the identified product group.

Also, the product identification device may perform product identification through a barcode learning model learned based on barcodes and a shape learning model learned based on shape information.

That is, the product identification device inputs the recognized barcode into the barcode learning model, inputs the recognized shape information into the shape learning model, combines the estimated result values from each of the barcode learning model and the shape learning model, and identifies the product. can do.

For example, the product identification device may identify a product through a result value estimated from a learning model having a recognition rate higher than a preset value among a barcode learning model and a shape learning model.

In addition, the product identification device may input images to the barcode learning model and the shape learning model a preset number of times, and identify the most aggregated result value among the estimated result values as the product.

10 is a flowchart illustrating a product identification method according to another embodiment of the present invention.

Referring to FIG. 10 , the product identification device may photograph a product through a slit formed between a plurality of conveyors (S310). Here, at least one product may include a plurality of barcodes printed with paint having a reflectance higher than a preset value for light in a specific wavelength range.

Here, barcodes of different shapes may be printed according to product categories. For example, barcode information may be printed in a “+”-shaped area in the case of fresh products, and barcode information may be printed in an “o”-shaped area in the case of industrial products.

Also, the barcodes may be printed at intervals set in advance according to product categories. For example, barcodes may be printed at intervals of 5 cm for fresh products and at intervals of 3 cm for industrial products.

In addition, barcodes may be printed in preset patterns according to product categories. For example, the barcode may be printed by arranging barcodes having different sizes in a predetermined pattern.

The barcode may be directly printed on the outer surface of the product or wrapping paper, or may be made in the form of a sheet or film and attached to the product.

In order to sense the above barcode, the product identification device may photograph the lower surface of the product through a camera capable of capturing a specific wavelength range in a state in which light of a specific wavelength range is irradiated.

Next, the product identification device may classify product categories based on the sensed barcode shape of at least one product (S320). Here, the product identification device may estimate the size of the barcode based on the number of barcodes recognized for a preset time from the time the product is recognized, and classify the product category according to the estimated size.

Also, the product identification device may classify product categories based on intervals between the plurality of barcodes. In this case, the barcodes may be spaced apart at predetermined intervals according to product categories and printed. For example, barcodes may be printed at intervals of 5 cm for fresh products and at intervals of 3 cm for industrial products.

Also, the product identification device may classify product categories based on a plurality of barcode print patterns. In this case, the barcode may be printed in a preset pattern according to the product category. For example, the barcode may be printed by arranging barcodes having different sizes in a predetermined pattern.

In addition, the product identification device may identify the product by searching for barcode information that matches the barcode information within the category classified in step S320.

11 is a flowchart illustrating a product identification method according to another embodiment of the present invention.

Referring to FIG. 11 , the product identification device may photograph the product through a plurality of cameras that detect light of different wavelength ranges (S410).

Here, the plurality of cameras may be two or more cameras selected from among a visible ray camera, an infrared camera, and a thermal imaging camera. However, it is not limited thereto, and the plurality of cameras may include at least one of a depth camera, a lidar, a radar, and an ultrasonic sensor.

Next, the product identification device may identify at least one product based on artificial intelligence (AI) based on shape information of at least one product included in a plurality of images taken from a plurality of cameras. (S420).

The product identification device may restore an image photographed by a visible ray camera to an image photographed by an infrared camera or a thermal imaging camera. However, it is not limited thereto, and the product identification device may generate an image with a high recognition rate through various combinations, such as restoring an image captured by an infrared camera to an image captured by a visible ray camera or a thermal imaging camera.

In addition, the product identification device includes a first learning model learned based on shape information of a product photographed by a visible ray camera and a second learning model learned based on shape information of a product photographed by an infrared camera. The product may be identified based on a third learning model learned based on shape information of the product photographed by the thermal image camera.

Here, the product identification device inputs images taken from cameras corresponding to each learning model to two or more learning models selected from among the first learning model, the second learning model, and the third learning model, and synthesizes the estimated result values, product can be identified.

In addition, the product identification device may identify a product by synthesizing result values estimated from learning models having a recognition rate higher than a preset value among result values estimated from the first learning model, the second learning model, and the third learning model. .

In addition, the product identification device inputs an image of a camera corresponding to each learning model to two or more learning models selected from among the first learning model, the second learning model, and the third learning model a preset number of times, and among the estimated result values, Highly aggregated result values can be identified as products.

Also, the product identification device may recognize a barcode in the captured image based on the image captured by the visible ray camera. Here, the product identification device may classify a product category based on an image captured by a thermal imaging camera, search for barcode information recognized within the category, and identify the product.

In addition, the product identification device may overlay a plurality of images taken from a plurality of cameras at the same location and identify the product based on the overlaid images. In addition, the product identification device may remove the background of the overlaid image, divide the foreground according to the outline, classify at least one product, and classify each product into one of irregular products and standard products. . That is, the product identification device may support clear distinction of outlines by overlaying a plurality of images taken at the same location from different types of cameras.

In addition, the product identification device may identify the product based on shape information of the product when the product is classified as an irregular product, and may identify the product based on the barcode of the product when classified as a standard product. That is, in the case of an irregular product, even if the barcode is printed on the packaging material, since the shape is irregular, the barcode recognition rate is remarkably low. so high Accordingly, the product identification unit 620 distinguishes irregular products from standard products, identifies products based on shape information in the case of irregular products having a relatively low barcode recognition rate, and barcodes in the case of standard products having a relatively high barcode recognition rate. Products can be identified through information.

12 is an exemplary diagram for explaining a product identification method according to an embodiment of the present invention.

Referring to FIG. 12, as shown in (A), a barcode in an image captured by a vision sensor is partially cut off, covered, twisted, or distorted, such as out of focus. Accordingly, there is a problem in that the accuracy of product recognition is lowered.

Accordingly, the product identification device identifies feature points in an image captured by a camera, restores a distorted barcode area, and identifies a product by recognizing information with the restored barcode area, thereby increasing the accuracy of product identification. can

That is, at least one product may be photographed through a product identification device camera, and a barcode area may be cropped in the photographed image. Here, cropping means an operation of removing an area other than a barcode area. Thereafter, the product identification device may identify at least one feature point in the cropped barcode area. The product identification device may restore a barcode area using feature points identified based on artificial intelligence (AI), and recognize information from the restored barcode area to identify a product.

13 is an exemplary diagram for explaining a product identification method according to another embodiment of the present invention.

Referring to FIG. 13 , the product identification device may recognize at least one product by dividing the product according to the foreground outline of the product in the captured image A.

In addition, the product identification device estimates the design of the product through the shape information of the product included in the photographed image, as in (C).

In addition, the product identification device may estimate a position where a barcode is expected to be located in the image based on the estimated design, as shown in (D).

And, as in (E), the product identification device calculates a distance value to the estimated position, focuses the camera based on the estimated position and the distance value to the estimated position, and performs zooming. can be filmed That is, since the product identification device is printed in a small size due to the nature of the barcode, when the image is focused on the overall shape of the product, the focus of the barcode is out of focus, and the barcode recognition rate may be significantly lowered. Accordingly, the product identification device can increase the barcode recognition rate by focusing on the estimated barcode position and re-recognizing the barcode through the recaptured image.

14 is an exemplary diagram for explaining a product identification method according to another embodiment of the present invention.

Referring to FIG. 14, the product identification device includes a first learning model learned based on shape information of a product photographed by a visible ray camera and learned based on shape information of a product photographed by an infrared camera. It may include a third learning model learned based on the second learning model and shape information of the product photographed by the thermal imaging camera.

The product identification device may input the first image, the second image, and the third image to the first learning model, the second learning model, and the third learning model, respectively. Here, the first image may be an image captured by a visible ray camera, the second image may be an image captured by an infrared camera, and the third image may be an image captured by a thermal imaging camera.

Here, the product identification device may identify a product by combining result values estimated from each learning model to two or more selected learning models among the first learning model, the second learning model, and the third learning model.

In addition, the product identification device may identify a product by synthesizing result values estimated from learning models having a recognition rate higher than a preset value among result values estimated from the first learning model, the second learning model, and the third learning model. .

In addition, the product identification device inputs an image of a camera corresponding to each learning model to two or more learning models selected from among the first learning model, the second learning model, and the third learning model a preset number of times, and among the estimated result values, Highly aggregated result values can be identified as products.

As such, the product identification method according to another embodiment of the present invention can enhance the accuracy of product identification by complexly recognizing the shape of a product photographed by a plurality of cameras that detect light in different wavelength ranges.

As described above, although preferred embodiments of the present invention have been disclosed in the present specification and drawings, it is in the technical field to which the present invention belongs that other modified examples based on the technical spirit of the present invention can be implemented in addition to the embodiments disclosed herein. It is self-evident to those skilled in the art. In addition, although specific terms have been used in the present specification and drawings, they are only used in a general sense to easily explain the technical content of the present invention and help understanding of the present invention, but are not intended to limit the scope of the present invention. Accordingly, the foregoing detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be selected by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

100, 200: product identification device 305: communication unit
310: input/output unit 315: storage unit
320: cropping unit 325: feature point identification unit
330: barcode restoration unit 335: product identification unit
405: communication unit 410: input/output unit
415: storage unit 420: barcode and shape information recognition unit
425: product identification unit 505: communication unit
510: input/output unit 515: storage unit
520: category classification unit 525: product identification unit
605: communication unit 610: input/output unit
615: storage unit 620: product identification unit

Claims (10)

Step, by the recognizing device, photographing at least one product through a camera;
Recognizing, by the recognizing device, a barcode in the captured image and shape information of a product included in the captured image based on artificial intelligence (AI); and
characterized in that it comprises; identifying a product included in the image based on the barcode and the shape information by the recognizing device;
The shooting step is
Characterized in that the at least one product is photographed through a plurality of cameras having different photographing positions,
At the stage of filming
The plurality of cameras are characterized in that for measuring the image of the at least one product and the depth of the product included in the image,
The shooting step is
Based on the shape information of the product included in the photographed image, a position where the barcode is expected to be located in the image is estimated, and a distance value to the estimated position is calculated.
The shooting step is
It is characterized by using a pre-learned classifier to estimate a region having shape information matching predefined barcode formation information for a product as a position where a barcode is expected to be located in the image,
The shooting step is
Based on the estimated position and the distance value to the estimated position, at least one of the plurality of cameras is focused and zoomed to take a picture,
The step of recognizing
Characterized in that recognizing the barcode in the photographed image, product identification method.
The method of claim 1, wherein the recognizing step
Recognizing the at least one product using the plurality of cameras, characterized in that product recognition is performed based on an image taken by a camera having the highest product recognition rate among images taken by the plurality of cameras , how to identify the product.
2. The method of claim 1, wherein the identifying step
Characterized in that, the product identification method of classifying the product group of the product included in the photographed image using a pre-learned classifier.
4. The method of claim 3, wherein the identifying step
The height value of the product included in the image is measured through distance information to the product included in the image, and the measured height value is compared with the average height value of the classified product group, so that the measured height value is If it is higher than the average height value of the product group, it is characterized in that it is determined that the products included in the image overlap.
5. The method of claim 4, wherein the identifying step
Characterized in that, the product identification method to search for the barcode in the product group (group) identified on the basis of the shape information.
5. The method of claim 4, wherein in the identifying step
The artificial intelligence comprises a barcode learning model learned based on barcodes and a shape learning model learned based on shape information, product identification method.
7. The method of claim 6, wherein the identifying step
The recognized barcode is input into the barcode learning model, the recognized shape information is input into the shape learning model, and result values estimated from each of the barcode learning model and the shape learning model are combined to identify the product. Characterized in that, product identification method.
memory;
transceiver; and
In combination with a computing device configured to include a processor for processing instructions resident in the memory,
Step, by the processor, photographing at least one product through a camera;
recognizing, by the processor, a barcode in the captured image and shape information of a product included in the captured image based on artificial intelligence (AI); and
By the processor, identifying a product included in the image based on the barcode and the shape information;
The shooting step is
Characterized in that the at least one product is photographed through a plurality of cameras having different photographing positions,
At the stage of filming
The plurality of cameras are characterized in that for measuring the image of the at least one product and the depth of the product included in the image,
The shooting step is
Characterized in that based on the shape information of the product included in the photographed image, a position where the barcode is expected to be located in the image is estimated, and a distance value to the estimated position is calculated,
The shooting step is
It is characterized by using a pre-learned classifier to estimate a region having shape information matching predefined barcode formation information for a product as a position where a barcode is expected to be located in the image,
The shooting step is
Based on the estimated position and the distance value to the estimated position, at least one of the plurality of cameras is focused and zoomed to take a picture,
The step of recognizing
A computer program recorded on a recording medium, characterized in that for re-recognizing the barcode in the captured image.
The method of claim 8, wherein the recognizing step
Recognizing the at least one product using the plurality of cameras, characterized in that product recognition is performed based on an image taken by a camera having the highest product recognition rate among images taken by the plurality of cameras , A computer program recorded on a recording medium.
10. The method of claim 9, wherein the identifying step
A computer program recorded on a recording medium, characterized in that the product group of the product included in the photographed image is classified using a pre-learned classifier.

Images