WO2022035141A1 - Method and system for verifying whether barcode is scanned - Google Patents

Abstract

A method for verifying whether a barcode is scanned, according to an embodiment of the present invention, comprises the steps of: a) obtaining, by an image obtaining module, an image of a product payment area including a barcode scanning area for product recognition and a product placement area in which a product to be barcode-scanned is placed; b) extracting, by an image-based product recognition module, a product object image in the barcode scanning area from the image, and recognizing a product being barcode scanned, on the basis of the product object image; c) calculating, by a total volume change amount calculation module, a total volume change amount of the barcode-scanned product in the product placement area on the basis of the image; and d) determining, by a barcode scanning error determination module, abnormal barcode scanning or barcode scanning omission of the product on the basis of the total volume change amount, a product recognized from the product object image, and a product recognized by the barcode scanning.

Description

Method and system for verifying whether barcode is scanned

The present invention relates to a method and system for verifying whether a barcode is scanned.

Most of the calculations of products purchased by customers in stores such as supermarkets and discount stores are automatically calculated by reading the barcode attached to the product. The POS system is connected to a computer, and a very useful system is widely used because it calculates the price of a product purchased by a customer very quickly and accurately. In this POS system, the price of each product is calculated by reading information from the barcode attached to the product through an input device called a scanner. Therefore, products not read by the scanner are excluded from the calculation, which is an important factor in the loss of sales of the store.

Recently, unmanned checkout counters (SCO, Self-Check Out) have been installed in each store, and customers directly scan barcodes at these unmanned checkout counters. In this process, barcode scan omissions or other product barcodes may be scanned.

In each store, a manager is usually assigned to monitor these actions, but as one manager monitors several unmanned checkout counters, it is practically difficult to monitor all of the above actions, and having several supervisors can increase labor costs. burden may increase.

Accordingly, the need for technology development capable of monitoring and preventing barcode scanning omissions or barcodes of other products in the payment manager through unmanned checkout counters (SCO, Self-Check Out) is gradually increasing.

An embodiment of the present invention can determine abnormal barcode scan or barcode scan omission of a product based on the total volume change of the product calculated using the depth image, the product recognized using the color image, and the product recognized by the barcode scan. An object of the present invention is to provide a method and system for verifying whether a barcode is scanned or not.

On the other hand, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned are clearly to those of ordinary skill in the art to which the present invention belongs from the description below. can be understood

A barcode scan verification method according to an embodiment of the present invention includes: a) an image of a product payment area including a barcode scanning area for product recognition and a product holding area in which barcode-scanned products are mounted through an image acquisition module obtaining a; b) extracting a product object image from the barcode scan area from the image through an image-based product recognition module, and recognizing a product under barcode scanning based on the product object image; c) calculating, through the total volume change amount calculation module, the total volume change amount of the product for which barcode scanning in the product holding area is completed based on the image; and d) through the barcode scan error determination module, determining the abnormal barcode scan or barcode scan omission of the product based on the total volume change of the product, the product recognized from the product object image, and the product recognized by the barcode scan includes

The step d) includes: d-1) determining whether a new product is placed in the product holding area by comparing the total volume change amount with a set reference volume change amount through a new product placement determination unit; d-2) determining, through the barcode scan determination unit, whether barcode scanning is performed on the new product before the new product is placed; and d-3) through the barcode scan error determining unit, when it is determined that the barcode scan has not been performed before the new product is placed in the product holding area, determining that the barcode scan is missing.

The step d) includes: d-1) determining whether a new product is placed in the product holding area by comparing the total volume change amount with a set reference volume change amount through a new product placement determination unit; d-2) determining, through the barcode scan determination unit, whether barcode scanning is performed on the new product before the new product is placed; and d-4) through the barcode scan error determining unit, when barcode scanning for the new product is performed, but the product recognized based on the product object image and the product recognized by the barcode scan do not match, the abnormal barcode It may include the step of determining with a scan.

The step a) includes: a-1) acquiring a depth image of the product holding area by a depth camera included in the image acquisition unit; a-2) acquiring a color image for the barcode scan area by a digital camera included in an image acquisition unit; a-3) extracting a motion region of an object from the color image through a motion region extraction unit; and a-4) determining, through the motion degree determination unit, whether the degree of motion of the object in the motion region is less than a set motion reference value, wherein step b) determines whether the degree of motion in step a-4) is It may be performed when it is less than the motion reference value.

The step c) includes: c-1) through the difference image acquisition unit, a depth difference image is obtained by comparing the depth image obtained in the product holding area with the background depth image, and the color image obtained in the product holding area is Comparing the background color image to obtain a color difference image; c-2) through the binarized image obtaining unit, binarizing the depth difference image to obtain a depth binarized image, and binarizing the color difference image to obtain a color binarized image; c-3) extracting a first image region having a first pixel value from the depth binarized image through a first image region extraction unit; c-4) extracting, through a second image region extraction unit, a second image region having a second pixel value from the color binarized image; c-5) calculating the first partial volume change of the first image region based on the area of the first image region and the depth image corresponding to the first image region through the first partial volume change calculation unit ; c-6) extracting a third image region that does not overlap the first image region from among the second image region through a third image region extraction unit; c-7) calculating a second partial volume change amount by integrating a reference constant with respect to the area of the third image region through a second partial volume change amount calculating unit; and c-8) using the total volume change calculation unit, calculating the total volume change amount of the product based on the first partial volume change amount and the second partial volume change amount.

e) updating the color image and depth image obtained after the product is mounted in the product holding area to the background color image and the background depth image when it is determined that normal barcode scanning has been performed through the background image update module may further include.

f) through the purchase list update module, when it is determined that the normal barcode scan has been performed, the step of adding the product identified by the normal barcode scan to the purchase list may be further included.

A recording medium according to another embodiment of the present invention is a computer-readable recording medium in which a program for executing a barcode scan or not verification method is recorded.

A barcode scan or not verification system according to another embodiment of the present invention includes: an image acquisition module for acquiring an image of a product payment area including a barcode scanning area for product recognition and a product holding area on which barcode-scanned products are mounted; an image-based product recognition module for extracting a product object image from the barcode scan area from the image and recognizing a product under barcode scanning based on the product object image; a total volume change calculation module for calculating a total volume change amount of the product for which barcode scanning in the product holding area is completed based on the image; and a barcode scan error determination module comprising determining an abnormal barcode scan or barcode scan omission of a product based on the total volume change of the product, the product recognized from the product object image, and the product recognized by the barcode scan do.

The barcode scan error determination module includes: a new product placement determination unit configured to compare the total volume change amount with a set reference volume change amount to determine whether a new product is placed in the product placement area; a barcode scan determination unit that determines whether barcode scanning is performed on the new product before the new product is placed; and a barcode scan error determining unit that determines that the barcode scan is missing when it is determined that barcode scanning is not performed before the new product is placed in the product holding area.

The barcode scan error determination module includes: a new product placement determination unit configured to compare the total volume change amount with a set reference volume change amount to determine whether a new product is placed in the product placement area; a barcode scan determination unit that determines whether barcode scanning is performed on the new product before the new product is placed; and a barcode scan error determination unit that determines the abnormal barcode scan when barcode scanning for the new product is performed, but the product recognized based on the product object image and the product recognized by the barcode scan do not match. can

The image acquisition module includes: an image acquisition unit that acquires a depth image of the product holding area by a depth camera, and acquires a color image of the barcode scan region by a digital camera; a motion region extraction unit for extracting a motion region of an object from the color image; and a motion degree determination unit that determines whether the degree of motion of the object in the motion region is less than a set motion reference value, and generates an operation signal of the image-based product recognition module when the degree of motion is less than the motion reference value.

The total volume change calculation module is configured to: obtain a depth difference image by comparing the depth image and the background depth image obtained in the product holding area, and compare the color image and the background color image obtained in the product holding area to obtain a color difference image a difference image acquisition unit that acquires a binarized image obtaining unit for binarizing the depth difference image to obtain a depth binarized image, and binarizing the color difference image to obtain a color binarized image; a first image region extraction unit for extracting a first image region having a first pixel value from the depth binarization image; a second image region extraction unit for extracting a second image region having a second pixel value from the color binarized image; a first partial volume change amount calculating unit for calculating a first partial volume change amount of the first image area based on an area of the first image area and a depth image corresponding to the first image area; a third image region extraction unit for extracting a third image region that does not overlap the first image region from among the second image region; a second partial volume change amount calculating unit for calculating a second partial volume change amount by integrating a reference constant with respect to the area of the third image region; and a total volume change calculation unit for calculating the total volume change amount of the product based on the first partial volume change amount and the second partial volume change amount.

It may further include a background image update module for updating the color image and the depth image obtained after the product is mounted on the product holding area to the background color image and the background depth image when it is determined that the normal barcode scan has been performed there is.

When it is determined that the normal barcode scan has been performed, the method may further include a purchase list update module for updating the purchase list by adding the product identified by the normal barcode scanning to the purchase list.

The method and system for verifying whether a barcode is scanned according to an embodiment of the present invention is an abnormality of a product based on the total volume change of a product calculated using a depth image, a product recognized using a color image, and a product recognized by a barcode scan Barcode scans or barcode scan omissions can be determined.

On the other hand, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description. will be able

1 is a block diagram schematically showing the configuration of a barcode scan or not verification system according to an embodiment of the present invention.

2 is a flowchart of a method for verifying whether a barcode is scanned according to an embodiment of the present invention.

3 is a block diagram schematically illustrating the configuration of an image acquisition module.

4 is a flowchart illustrating the step S100 in more detail.

5 is a block diagram schematically illustrating the configuration of a total volume change amount calculation module.

6 is a flowchart illustrating the step S300 in more detail.

7 is a diagram schematically showing the configuration of a barcode scan error determination module.

8 is a flowchart of step S400 according to the first embodiment.

9 is a flowchart of step S400 according to the second embodiment.

Hereinafter, specific embodiments of the present invention will be described in detail with reference to the drawings.

In addition, in describing the present invention, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

Examples of the present invention are provided to more completely explain the present invention to those of ordinary skill in the art, and the following examples may be modified in various other forms, and the scope of the present invention is as follows It is not limited to an Example.

Rather, these examples are provided so that this disclosure will be more thorough and complete, and will fully convey the spirit of the invention to those skilled in the art.

In addition, in the following drawings, each configuration is exaggerated for convenience and clarity of description, and the same reference numerals in the drawings refer to the same elements. As used herein, the term “and/or” includes any one and all combinations of one or more of those listed items.

The terminology used herein is used to describe specific embodiments, not to limit the present invention.

As used herein, the singular form may include the plural form unless the context clearly dictates otherwise. Also, as used herein, “comprise” and/or “comprising” refers to specifying the presence of the recited shapes, numbers, steps, actions, members, elements, and/or groups thereof. will,

It does not exclude the presence or addition of one or more other shapes, numbers, movements, members, elements and/or groups.

1 is a block diagram schematically showing the configuration of a barcode scan or not verification system 10 according to an embodiment of the present invention.

Referring to FIG. 1 , the barcode scan verification system 10 includes an image acquisition module 100 , an image-based product recognition module 200 , a total volume change calculation module 300 , a barcode scan error determination module 400 , and a background It may include an image update module 500 and a purchase list update module 600 .

The image acquisition module 100 may acquire an image of a product payment area including a barcode scanning area for product recognition and a product holding area in which barcode-scanned products are mounted. The image-based product recognition module 200 may extract a product object image from the barcode scan area from the image, and recognize a product under barcode scanning based on the product object image. The total volume change calculation module 300 is the Based on the image, it is possible to calculate the total volume change of the product for which barcode scanning in the product holding area is completed. The barcode scan error determination module 400 may determine the abnormal barcode scan or barcode scan omission of the product based on the total volume change of the product, the product recognized from the product object image, and the product recognized by the barcode scan. When it is determined that the normal barcode scan is performed, the background image update module 500 may update the color image and depth image obtained after the product is mounted in the product holding area to the background color image and the background depth image. there is. When it is determined that the normal barcode scan is performed, the purchase list update module 600 may update the purchase list by adding the product identified by the normal barcode scan to the purchase list.

2 is a flowchart of a method (S10) of verifying whether a barcode has been scanned according to an embodiment of the present invention.

Referring to FIG. 2 , the barcode scan verification method ( S10 ) is, through the image acquisition module 100 , in a product payment area including a barcode scanning area for product recognition and a product placement area in which barcode-scanned products are mounted. Step (S100) of obtaining an image for the product, through the image-based product recognition module 200, extracting a product object image from the barcode scan area from the image, and recognizing a product under barcode scanning based on the product object image Step (S200), through the total volume change calculation module 300, calculating the total volume change amount of the product for which barcode scanning in the product holding area is completed based on the image (S300), barcode scan error determination module ( Through 400), determining the abnormal barcode scan or barcode scan omission of the product based on the total volume change of the product, the product recognized from the product object image, and the product recognized by the barcode scan (S400), background image Updating the color image and the depth image obtained after the product is mounted in the product holding area to the background color image and the background depth image when it is determined that the normal barcode scan has been performed through the update module 500 ( S500) and through the purchase list update module 600, when it is determined that the normal barcode scan has been performed, adding the product identified by the normal barcode scan to the purchase list (S600).

Hereinafter, the barcode scan verification method ( S10 ) through the barcode scan verification system 10 will be described in more detail.

3 is a block diagram schematically illustrating the configuration of the image acquisition module 100 .

Referring to FIG. 3 , the image acquisition module 100 may include an image acquisition unit 110 , a motion region extraction unit 120 , and a motion degree determination unit 130 .

With the introduction of unmanned checkout counters (SCO, Self-Check Out), buyers who want to purchase products choose the products they want to purchase from the counter, bring them to the unmanned checkout counter, and scan the barcode directly.

The unmanned checkout counter is provided with a product payment area including an area for placing products before barcode scanning, a barcode scanning area for product recognition through a barcode scanner, and a product storage area for products on which barcode scanning has been completed.

An image acquisition unit 110 capable of acquiring an image of the product payment area and an image of a product existing in the product payment area is provided in the area where the product payment area can be viewed.

The image acquisition unit 110 may acquire a depth image of the product holding area by a depth camera, and acquire a color image of the barcode scan area by a digital camera. Of course, in addition to this, a depth camera and a digital camera included in the image acquisition unit 110 may be provided to acquire a depth image and a color image over the entire area of the product payment area.

Cameras can be provided as depth cameras and digital cameras. As a depth camera, a photodiode (PD) type camera, a vidicon type camera, a triangulation type 3D laser scanner, a depth camera using a structured beam pattern, and TOF using the difference in the reflection time of infrared rays A (Time-Of-Flight) type depth camera or the like may be applied. However, the present invention is not limited thereto, and any type of depth camera that can obtain 3D distance information of a product may be applied. A digital camera can be applied regardless of type as long as it can acquire color images.

The motion region extraction unit 120 may extract the motion region of the object from the color image. When the color image for the barcode scan area is obtained by the digital camera, the motion area extraction unit 120 may extract a motion area, which is an area in which motion of an object exists, from the color image for the barcode scan area. Here, the object may be a product for which barcode scanning is in progress, and in this case, only an image in which the movement of the product object exists may be extracted through the movement area extraction unit 120 .

The motion degree determination unit 130 determines whether the motion degree of the object in the motion region is less than a set motion reference value, and when the motion degree is less than the motion reference value, to generate an operation signal of the image-based product recognition module 200 can

When a blurring effect occurs on the image due to the movement of the product object, the product recognition rate decreases. In order to extract the product object image from the image through the image-based product recognition module 200 and recognize the product based on the product object image, , an image above the quality required by the image-based product recognition module 200 is required. To this end, the movement degree determination unit 130 determines whether the movement degree of the product object in the movement area extracted by the movement area extraction unit 120 is less than a movement reference value. If the degree of movement of the product object is less than the motion reference value, it is determined that an image of a quality higher than that required by the image-based product recognition module 200 has been acquired, and an operation signal for the operation of the image-based product recognition module 200 can be generated. there is.

For example, considering a process in which a buyer takes a product to a barcode scanner and scans it, the degree of movement of the product object may be greater than or equal to the movement reference value while approaching or moving away from the barcode scanner. However, while the barcode attached to the product is scanned with the barcode scanner, the movement of the product object is reduced, so that the degree of movement of the product object may be less than the movement reference value. Through this, the image-based product recognition module 200 may recognize the product being scanned by the barcode based on the image of the product object during the time the product object is being scanned with the barcode through the barcode scanner.

4 is a flowchart illustrating the step S100 in more detail.

Referring to FIG. 4 , step S100 includes acquiring a depth image of the product mounting area by a depth camera included in the image acquiring unit 110 ( S110 ), and a digital camera included in the image acquiring unit 110 . obtaining a color image for the barcode scan area by (S120), extracting a motion area of an object from the color image through the motion area extraction unit 120 (S130), and a motion degree determination unit ( Through 130), determining whether the degree of movement of the object in the movement area is less than a set movement reference value (S140) may be included.

In step S200, through the image-based product recognition module 200, a product object image may be extracted from the barcode scan area from the image, and a product under barcode scanning may be recognized based on the product object image.

As mentioned above, the image-based product recognition module 200 may operate only when an operation signal is generated from the movement degree determination unit 130 .

Image information of each product sold in a store is stored in the database. The image-based product recognition module 200 may recognize a product under barcode scanning by comparing image information of each product stored in the database with an image of a product object extracted from an image obtained from the barcode scanning area.

In addition, the image-based product recognition module 200 performs machine learning by using image information of each product as learning data, not image information of each product stored in the database, and then in the barcode scan area based on the learning result. From the product object image extracted from the acquired image, the product under barcode scanning can be recognized. In this case, it is also possible that machine learning is performed in a deep learning method through deep neural networks.

5 is a block diagram schematically illustrating the configuration of the total volume change amount calculation module 300 .

Referring to FIG. 5 , the total volume change calculation module 300 includes a difference image acquisition unit 310 , a binarized image acquisition unit 320 , a first image region extraction unit 330 , and a second image region extraction unit 340 . , a first partial volume change calculation unit 350 , a third image region extraction unit 360 , a second partial volume change amount calculation unit 370 , and a total volume change amount calculation unit 380 .

The difference image obtaining unit 310 compares the depth image and the background depth image obtained in the product holding area to obtain a depth difference image, and compares the color image obtained in the product mounting area with the background color image to obtain a color difference image can be obtained. The binarized image obtaining unit 320 may binarize the depth difference image to obtain a depth binarized image, and may binarize the color difference image to obtain a color binarized image. The first image region extraction unit 330 may extract a first image region having a first pixel value from the depth binarized image. The second image region extraction unit 340 may extract a second image region having a second pixel value from the color binarized image. The first partial volume change calculation unit 350 may calculate a first product volume change in the first image region based on the first image region and a depth image corresponding to the first image region. The third image region extraction unit 360 may extract a third image region that does not overlap the first image region from among the second image region. The second partial volume change calculation unit 370 may calculate a second product volume change amount by integrating a reference constant with respect to the third image region. The total volume change calculation unit 380 may calculate the total volume change amount of the product based on the first partial volume change amount and the second product volume change amount.

In more detail about the difference image obtaining unit 310 , the difference image obtaining unit 310 may obtain a depth difference image and a color difference image by using the background depth image and the background color image stored in the database. That is, the depth difference image is obtained by excluding the background depth image for the product mounting area from the depth image in the product mounting area acquired by the depth camera, and product placement is obtained from the color image in the product mounting area acquired by the digital camera. A color difference image may be obtained by excluding the background color image for the region.

As mentioned above, the binarized image obtaining unit 320 may binarize the depth difference image to obtain a depth binarized image, and may binarize the color difference image to obtain a color binarized image.

The first image region extracting unit 330 may extract a first image region having a first pixel value from the depth binarized image obtained through the binarized image obtaining unit 320 , and the second image region extraction unit 340 . may extract the second image region having the second pixel value from the color binarized image obtained through the binarized image obtaining unit 320 .

For example, a part of the object mounted on the product holding area is a part (first part) capable of height detection through the depth camera, and the remaining part of the same object is a part (second part) in which height detection is impossible through the depth camera. ) can be

In this case, the first image region extraction unit 330 extracts the first image region having the first pixel value from the depth binarization image obtained by binarizing the depth difference image. It may correspond to a part (the first part) capable of height detection through a depth camera among the mounted objects.

In addition, the second image region extraction unit 340 extracts a second image region having a second pixel value from the color binarized image obtained by binarizing the color difference image. It may correspond to the entire part of the mounted object.

The third image region extraction unit 360 extracts only the third image region that does not overlap the first image region among the second image regions. That is, the third image area may correspond to a portion (the second portion) in which the height of the object mounted on the product placement area cannot be detected through the depth camera.

The first partial volume change calculation unit 350 is configured to calculate the first partial volume, which is the volume change amount of the first portion corresponding to the first image region, based on the area of the first image region and the height of the first image region obtained through the depth image. change can be calculated.

The second partial volume change calculation unit 370 may calculate the second partial volume change amount, which is the volume change amount of the second part corresponding to the third image area, by integrating the reference constant with respect to the area of the third image area. In this case, the reference constant may be adjusted according to a user's setting. Since it is impossible to measure the height of the second part through the depth image, in the present invention, a reference constant is assumed to be the height of the second part and the volume of the second part is measured.

The total volume change calculation unit 380 calculates the total volume change amount of the product placed in the product placement area by summing the first partial volume change amount and the second partial volume change amount.

For example, it may be assumed that the customer puts a new product in the product holding area while several products are not already mounted in the product holding area. In this case, as exemplified above, the new product may be a product including both a part (first part) capable of height detection by the depth camera and a part (second part) in which height detection is not possible through the depth camera.

In this case, the first partial volume change calculation unit 350 calculates the volume change amount due to the part (first part) capable of height detection through the depth camera in the new product, that is, the first partial volume change amount.

In addition, the second partial volume change calculation unit 370 calculates the volume change amount due to the part (the second part) that cannot be detected in height through the depth camera in the new product, that is, the second partial volume change amount.

And the total volume change calculation unit 380 calculates the total volume change amount by adding the first partial volume change amount and the second partial volume change amount, and through this, the total volume change amount generated by additionally placing a new product in the product holding area is calculated can be

6 is a flowchart illustrating the step S300 in more detail.

Referring to FIG. 6 , in step S300 , a depth difference image is obtained by comparing the depth image obtained in the product holding area with the background depth image through the difference image obtaining unit 310 , and a color image obtained in the product holding area Comparing the color difference image with the background color image to obtain a color difference image (S310), binarizing the depth difference image through the binarization image acquisition unit 320 to obtain a depth binarization image, and binarizing the color difference image to color binarization Obtaining an image (S320), extracting a first image region having a first pixel value from the depth binarized image through a first image region extraction unit 330 (S330), A second image region extraction unit ( 340) extracting a second image region having a second pixel value from the color binarized image (S340), and the first image region and the first image region through a first partial volume change calculation unit 350 Calculating the first partial volume change amount of the first image region based on the depth image corresponding to ( S350 ), the first image region and the first image region among the second image region through the third image region extraction unit 360 Extracting a third non-overlapping image region ( S360 ), and calculating a second partial volume change amount by integrating a reference constant with respect to the third image region through the second partial volume change amount calculating unit 370 ( S370 ) ) and calculating the total volume change of the product based on the first partial volume change and the second product volume change through the total volume change calculation unit 380 ( S380 ).

7 is a diagram schematically showing the configuration of the barcode scan error determination module 400 .

Referring to FIG. 7 , the barcode scan error determination module 400 may include a new product placement determination unit 410 , a barcode scan determination unit 420 , and a barcode scan error determination unit 430 .

The new product placement determination unit 410 may compare the total volume change amount with a set reference volume change amount to determine whether the new product is placed in the product placement area. The barcode scan determination unit 420 may determine whether barcode scanning is performed on the new product before the new product is mounted. When it is determined that the barcode scan is not performed before the new product is placed in the product holding area, the barcode scan error determination unit 430 may determine that the barcode scan is missing. Also, when the product recognized based on the product object image and the product recognized by scanning the barcode do not match, the barcode scan error determining unit 430 may determine the abnormal barcode scan.

8 is a flowchart of step S400 according to the first embodiment.

Referring to FIG. 8 , according to the first embodiment of step S400, in step S400, the total volume change is compared with a set reference volume change amount through the new product placement determination unit 410, and a new product is placed in the product holding area. Determining whether the new product has been placed (S410), through the barcode scanning or not determining unit 420, determining whether barcode scanning has been performed on the new product before the new product is placed (S420), and a barcode scanning error determining unit ( Through 430), when it is determined that barcode scanning is not performed before the new product is placed in the product holding area, determining that the barcode scan is missing (S430) may include.

9 is a flowchart of step S400 according to the second embodiment.

Referring to FIG. 9 , according to the second embodiment of step S400, in step S400, the new product is placed in the product holding area by comparing the total volume change with a set reference volume change amount through the new product placement determination unit 410. Determining whether the new product has been placed (S410), through the barcode scanning or not determining unit 420, determining whether barcode scanning has been performed on the new product before the new product is placed (S420), and a barcode scanning error determining unit ( 430), when barcode scanning for the new product is performed, but the product recognized based on the product object image and the product recognized by the barcode scan do not match, determining the abnormal barcode scan (S440) may include

The first embodiment of the step S400 and the second embodiment of the step S400 performed through the barcode scan error determination module 400 are summarized as follows.

The new product placement determination unit 410 compares the total volume change amount in the product placement area obtained through the total volume change amount calculation module 300 with the reference volume change amount to determine whether the new product is placed in the product placement area. More specifically, when the total volume change exceeds the reference volume change amount, the new product placement determination unit 410 may determine that the new product is placed in the product placement area. In this case, the reference volume change amount may be set to correspond to the minimum volume of the product sold in the store. However, the present invention is not limited thereto and the reference volume change amount may be adjusted according to the user's selection.

The barcode scanning determination unit 420 determines whether the barcode scan of the new product is performed by the barcode scanner before the new product is mounted in the product placement area. In detail, when it is determined that the new product is placed in the product placement area through the new product placement determination unit 410, it is detected whether barcode scanning by the barcode scanner is performed at a point in time before it is determined that the new product is placed By doing so, it can be determined whether barcode scanning of the new product has been performed.

According to the first embodiment of the barcode scan error determination unit 430, the barcode scan error determination unit 430 determines that the new product has been placed through the new product placement determination unit 410, and the barcode scan error determination unit ( 420), when it is determined that the barcode scan for the new product has not been performed, it is determined that the barcode scan for the new product is omitted.

For example, when the customer moves the product to the product holding area without performing barcode scanning, the barcode scan error determining unit 430 may determine that the customer has missed scanning the barcode.

According to the second embodiment of the barcode scan error determination unit 430, the barcode scan error determination unit 430 determines that the new product has been placed through the new product placement determination unit 410, and the barcode scan error determination unit ( 420), when it is determined that the barcode scan for the new product has been performed, but the product recognized based on the product object image and the product recognized by the barcode scan do not match, the product to be reflected on the image acquisition unit 110 is It is determined that a barcode scan of a product other than that has been performed, that is, an abnormal barcode scan has been performed.

For example, when the customer places a low-priced product under the high-priced product in a state where the high-priced product is projected on the camera and only scans the barcode on the low-priced product, the barcode scan error determination unit 430 may It may be determined that the customer performed an abnormal barcode scan.

In the case of the above two embodiments of the barcode scan error determining unit 430, the barcode scan error determining unit 430 may stop an additional barcode scanning operation, furthermore, a communication signal indicating that the barcode scan is missing or abnormal barcode scanning has been performed; The manager can be notified by means of a light emission or an alarm.

According to another embodiment of the barcode scan error determination unit 430 , the barcode scan error determination unit 430 may determine the volume of the product recognized based on the product object image or the volume and total volume change of the product recognized by barcode scanning. It may be determined whether there is another type of barcode scanning error by comparing the total volume change calculated through the calculation module 300 .

More specifically, the database may store the volume of each product sold in the store, and accordingly, the product recognized based on the product object image or the product recognized by barcode scanning can be matched with the product volume stored in the database. can The barcode scan error determination unit 430 is configured to determine if the volume of the product recognized based on the product object image or the product recognized by the barcode scan and the volume change calculated by the total volume change calculation module 300 are different from each other, the product object It may be determined that the product recognized based on the image or the product recognized by barcode scanning has been moved to the product holding area with another product hidden under the product.

For example, after the customer hides the high-priced product under the low-priced product, only the low-priced product is illuminated on the image acquisition unit 110 and only barcode scans for the low-priced product are performed, and then the low-priced object and the high-priced object can be assumed to be mounted in the product storage area at the same time. In this case, the barcode scan error determination unit 430 calculates the total volume change calculated through the total volume change calculation module 300 compared to the volume of the product recognized based on the product object image or the volume of the product recognized by the barcode scan. can be determined to be larger, and through this, it can be determined that barcode scan omissions for expensive products have occurred.

Referring back to FIGS. 1 and 2 , the background image update module 500 performs normal barcode scanning rather than abnormal barcode scanning, barcode scan omission, and another type of barcode scanning error in the barcode scanning of new products through the barcode scanner. decide whether If it is determined that a normal barcode scan has been performed, the color image acquired after the new product is mounted in the product holding area is updated as a background color image, and the depth image acquired after the new product is mounted in the product holding area is displayed as a background depth image update to Through this, after the new product is placed in the product storage area, it can be determined whether another new product is placed in the product storage area.

When it is determined that the normal barcode scan is performed, the purchase list update module 600 performs an operation of adding the product identified by the normal barcode scan to the purchase list. Through this, all products scanned by the buyer can be registered in the purchase list.

The recording medium according to another embodiment of the present invention may be a computer-readable recording medium in which a program for executing the barcode scan or not verification method (S10) is recorded.

The method (S10) and system 10 for verifying whether barcode scanning according to an embodiment of the present invention is a total volume change of a product calculated using a depth image, a product recognized using a color image, and a product recognized by barcode scanning Based on this, it is possible to judge abnormal barcode scan or omission of barcode scan of the product.

Above, the micro light emitting device having an omnidirectional reflector structure and a manufacturing method thereof according to an embodiment of the present invention have been described as specific embodiments, but this is merely an example, and the present invention is not limited thereto, and the basic idea disclosed in the present specification is not limited thereto. should be construed as having the widest scope to follow. A person skilled in the art may implement a pattern of a shape not specified by combining or substituting the disclosed embodiments, but this also does not depart from the scope of the present invention. In addition, those skilled in the art can easily change or modify the disclosed embodiments based on the present specification, and it is clear that such changes or modifications also fall within the scope of the present invention.

Claims (15)

1. a) acquiring, through the image acquisition module, an image of a product payment area including a barcode scanning area for product recognition and a product holding area in which barcode-scanned products are mounted;

   b) extracting a product object image from the barcode scan area from the image through an image-based product recognition module, and recognizing a product under barcode scanning based on the product object image;

   c) calculating, through the total volume change amount calculation module, the total volume change amount of the product for which barcode scanning in the product holding area is completed based on the image; and

   d) through the barcode scan error determination module, determining the abnormal barcode scan or barcode scan omission of the product based on the total volume change of the product, the product recognized from the product object image, and the product recognized by the barcode scan How to verify whether a barcode has been scanned or not.
2. According to claim 1,

   Step d) is:

   d-1) determining whether a new product is placed in the product holding area by comparing the total volume change amount with a set reference volume change amount through a new product placement determination unit;

   d-2) determining, through the barcode scan determination unit, whether barcode scanning is performed on the new product before the new product is placed; and

   d-3) through the barcode scan error determination unit, when it is determined that the barcode scan has not been performed before the new product is placed in the product holding area, determining whether the barcode scan is missing; method.
3. According to claim 1,

   Step d) is:

   d-1) determining whether a new product is placed in the product holding area by comparing the total volume change amount with a set reference volume change amount through a new product placement determination unit;

   d-2) determining, through the barcode scan determination unit, whether barcode scanning is performed on the new product before the new product is placed; and

   d-4) If barcode scanning is performed on the new product through the barcode scan error determination unit, but the product recognized based on the product object image and the product recognized by the barcode scan do not match, scan the abnormal barcode A method of verifying whether a barcode is scanned, including determining as
4. According to claim 1,

   Step a) is:

   a-1) acquiring a depth image of the product holding area by a depth camera included in an image acquisition unit;

   a-2) acquiring a color image for the barcode scan area by a digital camera included in an image acquisition unit;

   a-3) extracting a motion region of an object from the color image through a motion region extraction unit; and

   a-4) through a motion degree determination unit, determining whether the motion degree of the object in the motion area is less than a set motion reference value;

   The step b) is a barcode scanning or not verification method performed when the degree of motion is less than the motion reference value in step a-4).
5. According to claim 1,

   Step c) is:

   c-1) Through the difference image acquisition unit, a depth difference image is obtained by comparing the depth image obtained in the product holding area with the background depth image, and the color image obtained in the product mounting area is compared with the background color image obtaining a color difference image;

   c-2) through a binarized image obtaining unit, binarizing the depth difference image to obtain a depth binarized image, and binarizing the color difference image to obtain a color binarized image;

   c-3) extracting a first image region having a first pixel value from the depth binarized image through a first image region extraction unit;

   c-4) extracting, through a second image region extraction unit, a second image region having a second pixel value from the color binarized image;

   c-5) calculating the first partial volume change of the first image region based on the area of the first image region and the depth image corresponding to the first image region through the first partial volume change calculation unit ;

   c-6) extracting, through a third image region extraction unit, a third image region that does not overlap the first image region from among the second image region;

   c-7) calculating, through a second partial volume change amount calculating unit, a second partial volume change amount by integrating a reference constant with respect to the area of the third image region; and

   c-8) through a total volume change calculation unit, based on the first partial volume change amount and the second partial volume change amount, calculating the total volume change amount of the product;
6. According to claim 1,

   e) updating the color image and depth image obtained after the product is placed in the product holding area to the background color image and the background depth image when it is determined that normal barcode scanning has been performed through the background image update module A method of verifying whether a barcode has been scanned further comprising a.
7. 7. The method of claim 6,

   f) through the purchase list update module, when it is determined that the normal barcode scan has been performed, adding the product identified by the normal barcode scan to the purchase list;
8. A computer-readable recording medium in which a program for executing the method for verifying whether or not to scan a barcode according to any one of claims 1 to 7 is recorded.
9. an image acquisition module for acquiring an image of a product payment area including a barcode scanning area for product recognition and a product holding area in which barcode-scanned products are mounted;

   an image-based product recognition module for extracting a product object image from the barcode scan area from the image, and recognizing a product under barcode scanning based on the product object image;

   a total volume change calculation module for calculating a total volume change amount of the product for which barcode scanning in the product holding area is completed based on the image; and

   A barcode scan error determination module comprising the step of determining an abnormal barcode scan or barcode scan omission of a product based on the total volume change of the product, the product recognized from the product object image, and the product recognized by the barcode scan Barcode scan verification system.
10. 10. The method of claim 9,

    The barcode scan error determination module includes:

    a new product placement determination unit that compares the total volume change amount with a set reference volume change amount and determines whether a new product is placed in the product placement area;

    a barcode scan determination unit that determines whether barcode scanning is performed on the new product before the new product is placed; and

    and a barcode scan error determining unit that determines that the barcode scan is missing when it is determined that the barcode scan is not performed before the new product is placed in the product holding area.
11. 10. The method of claim 9,

    The barcode scan error determination module includes:

    a new product placement determination unit that compares the total volume change amount with a set reference volume change amount and determines whether a new product is placed in the product placement area;

    a barcode scan determination unit that determines whether barcode scanning is performed on the new product before the new product is placed; and

    Barcode including a barcode scan error determination unit that determines the abnormal barcode scan when barcode scanning for the new product is performed, but the product recognized based on the product object image and the product recognized by the barcode scan do not match Scan Verification System.
12. 10. The method of claim 9,

    The image acquisition module includes:

    an image acquisition unit for acquiring a depth image of the product holding area by a depth camera and acquiring a color image of the barcode scanning area by a digital camera;

    a motion region extraction unit for extracting a motion region of an object from the color image; and

    and a motion degree determination unit that determines whether the degree of motion of the object in the motion area is less than a set motion reference value, and generates an operation signal of the image-based product recognition module when the degree of motion is less than the motion reference value. system.
13. 10. The method of claim 9,

    The total volume change calculation module includes:

    a difference image acquisition unit configured to obtain a depth difference image by comparing the depth image and the background depth image obtained in the product mounting area, and to obtain a color difference image by comparing the color image obtained in the product mounting area with the background color image;

    a binarized image obtaining unit for binarizing the depth difference image to obtain a depth binarized image, and binarizing the color difference image to obtain a color binarized image;

    a first image region extraction unit for extracting a first image region having a first pixel value from the depth binarization image;

    a second image region extraction unit for extracting a second image region having a second pixel value from the color binarized image;

    a first partial volume change amount calculating unit for calculating a first partial volume change amount of the first image area based on an area of the first image area and a depth image corresponding to the first image area;

    a third image region extraction unit for extracting a third image region that does not overlap the first image region from among the second image region;

    a second partial volume change amount calculating unit for calculating a second partial volume change amount by integrating a reference constant with respect to the area of the third image region; and

    and a total volume change calculation unit for calculating a total volume change amount of the product based on the first partial volume change amount and the second partial volume change amount.
14. 10. The method of claim 9,

    When it is determined that a normal barcode scan has been performed, the barcode further comprising a background image update module for updating the color image and depth image obtained after the product is mounted on the product holding area to the background color image and the background depth image Scan Verification System.
15. 15. The method of claim 14,

    and a purchase list update module for updating the purchase list by adding the product identified by the normal barcode scan to the purchase list when it is determined that the normal barcode scan has been performed.

Images